Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251024 19:47:51.514020 1011 test_util.cc:276] Using random seed: -751804556
2025-10-24T19:47:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T19:47:51Z Disabled control of system clock
I20251024 19:47:51.551504 1011 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskc9NInd/build/asan/bin/kudu
/tmp/dist-test-taskc9NInd/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.252.254:36505
--webserver_interface=127.0.252.254
--webserver_port=0
--builtin_ntp_servers=127.0.252.212:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251024 19:47:51.932165 1019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 19:47:51.932507 1019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 19:47:51.932637 1019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 19:47:51.941511 1019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 19:47:51.941651 1019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 19:47:51.941754 1019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 19:47:51.941831 1019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 19:47:51.952077 1019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.252.212:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.252.254:36505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.252.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 24 Oct 2025 19:43:25 UTC on 24a791456cd2
build id 8700
ASAN enabled
I20251024 19:47:51.954025 1019 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 19:47:51.956377 1019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 19:47:51.965406 1024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:47:51.965929 1025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:47:51.966405 1027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 19:47:51.967340 1019 server_base.cc:1047] running on GCE node
I20251024 19:47:51.968075 1019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 19:47:51.969436 1019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 19:47:51.970638 1019 hybrid_clock.cc:648] HybridClock initialized: now 1761335271970571 us; error 62 us; skew 500 ppm
I20251024 19:47:51.970957 1019 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251024 19:47:51.973196 1019 webserver.cc:492] Webserver started at http://127.0.252.254:42051/ using document root <none> and password file <none>
I20251024 19:47:51.973968 1019 fs_manager.cc:362] Metadata directory not provided
I20251024 19:47:51.974071 1019 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 19:47:51.974422 1019 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 19:47:51.976709 1019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data/instance:
uuid: "ef8906492b1a4368936fac95c11e1f73"
format_stamp: "Formatted at 2025-10-24 19:47:51 on dist-test-slave-53sn"
I20251024 19:47:51.977505 1019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal/instance:
uuid: "ef8906492b1a4368936fac95c11e1f73"
format_stamp: "Formatted at 2025-10-24 19:47:51 on dist-test-slave-53sn"
I20251024 19:47:51.982596 1019 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20251024 19:47:51.985788 1033 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 19:47:51.986964 1019 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251024 19:47:51.987133 1019 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
uuid: "ef8906492b1a4368936fac95c11e1f73"
format_stamp: "Formatted at 2025-10-24 19:47:51 on dist-test-slave-53sn"
I20251024 19:47:51.987375 1019 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 19:47:52.012022 1019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 19:47:52.013055 1019 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 19:47:52.013356 1019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 19:47:52.035233 1019 rpc_server.cc:307] RPC server started. Bound to: 127.0.252.254:36505
I20251024 19:47:52.035250 1084 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.252.254:36505 every 8 connection(s)
I20251024 19:47:52.036728 1019 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data/info.pb
I20251024 19:47:52.038831 1011 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskc9NInd/build/asan/bin/kudu as pid 1019
I20251024 19:47:52.039176 1011 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal/instance
I20251024 19:47:52.042331 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:52.042760 1011 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskc9NInd/build/asan/bin/kudu
/tmp/dist-test-taskc9NInd/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.252.253:41375
--webserver_interface=127.0.252.253
--webserver_port=0
--builtin_ntp_servers=127.0.252.212:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251024 19:47:52.050006 1085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } has no permanent_uuid. Determining permanent_uuid...
I20251024 19:47:52.066169 1085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } has no permanent_uuid. Determining permanent_uuid...
W20251024 19:47:52.067127 1035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.252.253:41375: connect: Connection refused (error 111)
W20251024 19:47:52.068797 1085 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.253:41375: Network error: Client connection negotiation failed: client connection to 127.0.252.253:41375: connect: Connection refused (error 111)
I20251024 19:47:52.118141 1085 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } attempt: 1
W20251024 19:47:52.119947 1085 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.253:41375: Network error: Client connection negotiation failed: client connection to 127.0.252.253:41375: connect: Connection refused (error 111)
I20251024 19:47:52.188220 1085 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } attempt: 2
W20251024 19:47:52.189980 1085 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.253:41375: Network error: Client connection negotiation failed: client connection to 127.0.252.253:41375: connect: Connection refused (error 111)
I20251024 19:47:52.281284 1085 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } attempt: 3
W20251024 19:47:52.283193 1085 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.253:41375: Network error: Client connection negotiation failed: client connection to 127.0.252.253:41375: connect: Connection refused (error 111)
W20251024 19:47:52.414232 1087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 19:47:52.414556 1087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 19:47:52.414657 1087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 19:47:52.424424 1087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 19:47:52.424557 1087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 19:47:52.424631 1087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 19:47:52.424718 1087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 19:47:52.426534 1085 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } attempt: 4
W20251024 19:47:52.428359 1085 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.253:41375: Network error: Client connection negotiation failed: client connection to 127.0.252.253:41375: connect: Connection refused (error 111)
I20251024 19:47:52.436111 1087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.252.212:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.252.253:41375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.252.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 24 Oct 2025 19:43:25 UTC on 24a791456cd2
build id 8700
ASAN enabled
I20251024 19:47:52.437743 1087 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 19:47:52.439930 1087 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 19:47:52.447832 1095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:47:52.447983 1096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:47:52.449316 1098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 19:47:52.449898 1087 server_base.cc:1047] running on GCE node
I20251024 19:47:52.450605 1087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 19:47:52.451972 1087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 19:47:52.453197 1087 hybrid_clock.cc:648] HybridClock initialized: now 1761335272453154 us; error 37 us; skew 500 ppm
I20251024 19:47:52.453550 1087 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251024 19:47:52.455751 1087 webserver.cc:492] Webserver started at http://127.0.252.253:43901/ using document root <none> and password file <none>
I20251024 19:47:52.456485 1087 fs_manager.cc:362] Metadata directory not provided
I20251024 19:47:52.456578 1087 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 19:47:52.456938 1087 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 19:47:52.459364 1087 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data/instance:
uuid: "82e82847f8904b59b03f446b7fce6233"
format_stamp: "Formatted at 2025-10-24 19:47:52 on dist-test-slave-53sn"
I20251024 19:47:52.460160 1087 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/wal/instance:
uuid: "82e82847f8904b59b03f446b7fce6233"
format_stamp: "Formatted at 2025-10-24 19:47:52 on dist-test-slave-53sn"
I20251024 19:47:52.466277 1087 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20251024 19:47:52.469926 1104 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 19:47:52.471266 1087 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251024 19:47:52.471436 1087 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data,/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/wal
uuid: "82e82847f8904b59b03f446b7fce6233"
format_stamp: "Formatted at 2025-10-24 19:47:52 on dist-test-slave-53sn"
I20251024 19:47:52.471654 1087 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 19:47:52.495712 1087 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 19:47:52.496650 1087 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 19:47:52.496951 1087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 19:47:52.522549 1087 rpc_server.cc:307] RPC server started. Bound to: 127.0.252.253:41375
I20251024 19:47:52.522730 1155 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.252.253:41375 every 8 connection(s)
I20251024 19:47:52.524106 1087 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data/info.pb
I20251024 19:47:52.528724 1156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:52.532876 1011 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskc9NInd/build/asan/bin/kudu as pid 1087
I20251024 19:47:52.533057 1011 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/wal/instance
I20251024 19:47:52.534200 1011 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskc9NInd/build/asan/bin/kudu
/tmp/dist-test-taskc9NInd/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.252.252:36627
--webserver_interface=127.0.252.252
--webserver_port=0
--builtin_ntp_servers=127.0.252.212:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251024 19:47:52.535671 1156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } has no permanent_uuid. Determining permanent_uuid...
I20251024 19:47:52.550237 1156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } has no permanent_uuid. Determining permanent_uuid...
I20251024 19:47:52.560729 1156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } has no permanent_uuid. Determining permanent_uuid...
W20251024 19:47:52.562107 1108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.252.252:36627: connect: Connection refused (error 111)
W20251024 19:47:52.563992 1156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.252:36627: Network error: Client connection negotiation failed: client connection to 127.0.252.252:36627: connect: Connection refused (error 111)
I20251024 19:47:52.613400 1156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } attempt: 1
W20251024 19:47:52.615979 1156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.252:36627: Network error: Client connection negotiation failed: client connection to 127.0.252.252:36627: connect: Connection refused (error 111)
I20251024 19:47:52.684303 1156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } attempt: 2
W20251024 19:47:52.686442 1156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.252:36627: Network error: Client connection negotiation failed: client connection to 127.0.252.252:36627: connect: Connection refused (error 111)
I20251024 19:47:52.727773 1085 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } attempt: 5
I20251024 19:47:52.735844 1085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } has no permanent_uuid. Determining permanent_uuid...
W20251024 19:47:52.738027 1085 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.252:36627: Network error: Client connection negotiation failed: client connection to 127.0.252.252:36627: connect: Connection refused (error 111)
I20251024 19:47:52.777658 1156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } attempt: 3
W20251024 19:47:52.779683 1156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.252:36627: Network error: Client connection negotiation failed: client connection to 127.0.252.252:36627: connect: Connection refused (error 111)
I20251024 19:47:52.789286 1085 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } attempt: 1
W20251024 19:47:52.791304 1085 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.252:36627: Network error: Client connection negotiation failed: client connection to 127.0.252.252:36627: connect: Connection refused (error 111)
I20251024 19:47:52.859563 1085 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } attempt: 2
W20251024 19:47:52.861703 1085 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.252:36627: Network error: Client connection negotiation failed: client connection to 127.0.252.252:36627: connect: Connection refused (error 111)
W20251024 19:47:52.910931 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 19:47:52.911337 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 19:47:52.911566 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 19:47:52.920002 1158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 19:47:52.920109 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 19:47:52.920164 1158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 19:47:52.920212 1158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 19:47:52.922935 1156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } attempt: 4
W20251024 19:47:52.924930 1156 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.252:36627: Network error: Client connection negotiation failed: client connection to 127.0.252.252:36627: connect: Connection refused (error 111)
I20251024 19:47:52.931332 1158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.252.212:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.252.252:36627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.252.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 24 Oct 2025 19:43:25 UTC on 24a791456cd2
build id 8700
ASAN enabled
I20251024 19:47:52.932982 1158 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 19:47:52.934787 1158 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 19:47:52.943075 1167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:47:52.943274 1166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:47:52.944072 1169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 19:47:52.944561 1158 server_base.cc:1047] running on GCE node
I20251024 19:47:52.945313 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 19:47:52.946725 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 19:47:52.947927 1158 hybrid_clock.cc:648] HybridClock initialized: now 1761335272947880 us; error 39 us; skew 500 ppm
I20251024 19:47:52.948285 1158 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251024 19:47:52.950712 1158 webserver.cc:492] Webserver started at http://127.0.252.252:42633/ using document root <none> and password file <none>
I20251024 19:47:52.951448 1158 fs_manager.cc:362] Metadata directory not provided
I20251024 19:47:52.951572 1158 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 19:47:52.951932 1158 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 19:47:52.954334 1158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/data/instance:
uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c"
format_stamp: "Formatted at 2025-10-24 19:47:52 on dist-test-slave-53sn"
I20251024 19:47:52.955098 1158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/wal/instance:
uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c"
format_stamp: "Formatted at 2025-10-24 19:47:52 on dist-test-slave-53sn"
I20251024 19:47:52.960501 1158 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20251024 19:47:52.964260 1175 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 19:47:52.965616 1158 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251024 19:47:52.965796 1158 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/data,/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/wal
uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c"
format_stamp: "Formatted at 2025-10-24 19:47:52 on dist-test-slave-53sn"
I20251024 19:47:52.966035 1158 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 19:47:52.967937 1085 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } attempt: 3
W20251024 19:47:52.969785 1085 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.252.252:36627: Network error: Client connection negotiation failed: client connection to 127.0.252.252:36627: connect: Connection refused (error 111)
I20251024 19:47:52.996551 1158 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 19:47:52.997732 1158 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 19:47:52.998032 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 19:47:53.025272 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.252.252:36627
I20251024 19:47:53.025291 1226 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.252.252:36627 every 8 connection(s)
I20251024 19:47:53.027091 1158 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/data/info.pb
I20251024 19:47:53.032368 1227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:53.035071 1011 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskc9NInd/build/asan/bin/kudu as pid 1158
I20251024 19:47:53.035280 1011 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-2/wal/instance
I20251024 19:47:53.040516 1227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } has no permanent_uuid. Determining permanent_uuid...
I20251024 19:47:53.057834 1227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } has no permanent_uuid. Determining permanent_uuid...
I20251024 19:47:53.066953 1227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } has no permanent_uuid. Determining permanent_uuid...
I20251024 19:47:53.079741 1227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Bootstrap starting.
I20251024 19:47:53.082819 1227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:53.083909 1227 log.cc:826] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Log is configured to *not* fsync() on all Append() calls
I20251024 19:47:53.086946 1227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: No bootstrap required, opened a new log
I20251024 19:47:53.091924 1227 raft_consensus.cc:359] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:47:53.092219 1227 raft_consensus.cc:385] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:53.092337 1227 raft_consensus.cc:740] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc8f71a5b70749a4a3e3a599bcb3d22c, State: Initialized, Role: FOLLOWER
I20251024 19:47:53.092967 1227 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:47:53.094122 1236 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:47:53.094439 1236 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: This master's current role is: FOLLOWER
I20251024 19:47:53.095161 1227 sys_catalog.cc:565] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: configured and running, proceeding with master startup.
W20251024 19:47:53.106956 1247 catalog_manager.cc:1568] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251024 19:47:53.107100 1247 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251024 19:47:53.103036 1158 master_runner.cc:186] Error getting master registration for 127.0.252.254:36505: OK, instance_id { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" instance_seqno: 1761335272029910 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251024 19:47:53.110406 1158 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251024 19:47:53.123087 1236 raft_consensus.cc:493] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:53.123380 1236 raft_consensus.cc:515] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:47:53.125216 1236 leader_election.cc:290] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef8906492b1a4368936fac95c11e1f73 (127.0.252.254:36505), 82e82847f8904b59b03f446b7fce6233 (127.0.252.253:41375)
I20251024 19:47:53.146948 1085 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } attempt: 4
I20251024 19:47:53.157789 1085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Bootstrap starting.
I20251024 19:47:53.162520 1085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:53.164848 1085 log.cc:826] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Log is configured to *not* fsync() on all Append() calls
I20251024 19:47:53.167903 1085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: No bootstrap required, opened a new log
I20251024 19:47:53.175307 1085 raft_consensus.cc:359] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:47:53.175621 1085 raft_consensus.cc:385] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:53.175737 1085 raft_consensus.cc:740] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8906492b1a4368936fac95c11e1f73, State: Initialized, Role: FOLLOWER
I20251024 19:47:53.176330 1085 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:47:53.177317 1250 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:47:53.177639 1250 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 19:47:53.178452 1085 sys_catalog.cc:565] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 19:47:53.178524 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8906492b1a4368936fac95c11e1f73" is_pre_election: true
I20251024 19:47:53.179075 1060 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc8f71a5b70749a4a3e3a599bcb3d22c in term 0.
I20251024 19:47:53.179715 1177 leader_election.cc:304] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc8f71a5b70749a4a3e3a599bcb3d22c, ef8906492b1a4368936fac95c11e1f73; no voters:
I20251024 19:47:53.180239 1236 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:47:53.180389 1236 raft_consensus.cc:493] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:47:53.180488 1236 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:53.184890 1236 raft_consensus.cc:515] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:47:53.186477 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8906492b1a4368936fac95c11e1f73"
I20251024 19:47:53.187005 1060 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:53.187239 1236 leader_election.cc:290] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [CANDIDATE]: Term 1 election: Requested vote from peers ef8906492b1a4368936fac95c11e1f73 (127.0.252.254:36505), 82e82847f8904b59b03f446b7fce6233 (127.0.252.253:41375)
I20251024 19:47:53.192625 1060 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc8f71a5b70749a4a3e3a599bcb3d22c in term 1.
I20251024 19:47:53.193357 1177 leader_election.cc:304] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc8f71a5b70749a4a3e3a599bcb3d22c, ef8906492b1a4368936fac95c11e1f73; no voters:
I20251024 19:47:53.193768 1236 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:47:53.194281 1236 raft_consensus.cc:697] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [term 1 LEADER]: Becoming Leader. State: Replica: dc8f71a5b70749a4a3e3a599bcb3d22c, State: Running, Role: LEADER
I20251024 19:47:53.194841 1236 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:47:53.198716 1256 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc8f71a5b70749a4a3e3a599bcb3d22c. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:47:53.199100 1256 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: This master's current role is: LEADER
I20251024 19:47:53.207890 1262 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 19:47:53.210243 1262 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 19:47:53.224064 1156 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } attempt: 5
I20251024 19:47:53.221827 1019 master_runner.cc:186] Error getting master registration for 127.0.252.253:41375: OK, instance_id { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" instance_seqno: 1761335272516781 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251024 19:47:53.229071 1019 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251024 19:47:53.232213 1060 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Refusing update from remote peer dc8f71a5b70749a4a3e3a599bcb3d22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 19:47:53.233103 1256 consensus_queue.cc:1048] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:47:53.242045 1156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233: Bootstrap starting.
I20251024 19:47:53.249044 1156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233: Neither blocks nor log segments found. Creating new log.
W20251024 19:47:53.250440 1181 tablet.cc:2378] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 19:47:53.251029 1156 log.cc:826] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233: Log is configured to *not* fsync() on all Append() calls
W20251024 19:47:53.254109 1265 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251024 19:47:53.254791 1265 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251024 19:47:53.266711 1039 tablet.cc:2378] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 19:47:53.278121 1156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233: No bootstrap required, opened a new log
I20251024 19:47:53.287428 1250 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc8f71a5b70749a4a3e3a599bcb3d22c. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:47:53.287791 1250 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 19:47:53.290875 1156 raft_consensus.cc:359] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:47:53.291318 1156 raft_consensus.cc:385] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:53.291479 1156 raft_consensus.cc:740] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e82847f8904b59b03f446b7fce6233, State: Initialized, Role: FOLLOWER
I20251024 19:47:53.292268 1156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:47:53.300124 1263 mvcc.cc:204] Tried to move back new op lower bound from 7214429279108222976 to 7214429279011164160. Current Snapshot: MvccSnapshot[applied={T|T < 7214429279108222976}]
I20251024 19:47:53.301772 1156 sys_catalog.cc:565] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 19:47:53.302091 1131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82e82847f8904b59b03f446b7fce6233" is_pre_election: true
I20251024 19:47:53.302659 1131 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc8f71a5b70749a4a3e3a599bcb3d22c in term 0.
I20251024 19:47:53.299482 1270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:47:53.305847 1129 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:53.307009 1256 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:47:53.307246 1256 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: This master's current role is: LEADER
I20251024 19:47:53.305997 1130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82e82847f8904b59b03f446b7fce6233"
I20251024 19:47:53.311614 1129 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [term 1 FOLLOWER]: Refusing update from remote peer dc8f71a5b70749a4a3e3a599bcb3d22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 19:47:53.312644 1270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 19:47:53.312644 1236 consensus_queue.cc:1048] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:47:53.313647 1262 catalog_manager.cc:1357] Generated new cluster ID: 57054fc705ad43a6b8ba5fe9c17ff20a
I20251024 19:47:53.313757 1262 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 19:47:53.328199 1250 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:47:53.328447 1250 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 19:47:53.338711 1282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:47:53.339107 1282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 19:47:53.339529 1270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc8f71a5b70749a4a3e3a599bcb3d22c. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:47:53.339757 1270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 19:47:53.340945 1262 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 19:47:53.342078 1236 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:47:53.350284 1236 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: This master's current role is: LEADER
I20251024 19:47:53.353380 1262 catalog_manager.cc:1514] Loading token signing keys...
I20251024 19:47:53.364195 1288 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251024 19:47:53.378710 1262 catalog_manager.cc:6022] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Generated new TSK 0
I20251024 19:47:53.380000 1262 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 19:47:53.406082 1011 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskc9NInd/build/asan/bin/kudu
/tmp/dist-test-taskc9NInd/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.252.193:0
--local_ip_for_outbound_sockets=127.0.252.193
--webserver_interface=127.0.252.193
--webserver_port=0
--tserver_master_addrs=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--builtin_ntp_servers=127.0.252.212:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20251024 19:47:53.823697 1289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 19:47:53.824080 1289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 19:47:53.824232 1289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 19:47:53.834167 1289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 19:47:53.834379 1289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.252.193
I20251024 19:47:53.845381 1289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.252.212:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.252.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.252.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.252.193
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 24 Oct 2025 19:43:25 UTC on 24a791456cd2
build id 8700
ASAN enabled
I20251024 19:47:53.847133 1289 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 19:47:53.848915 1289 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 19:47:53.859303 1294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:47:53.860029 1295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:47:53.862416 1297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 19:47:53.862776 1289 server_base.cc:1047] running on GCE node
I20251024 19:47:53.863739 1289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 19:47:53.864955 1289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 19:47:53.866180 1289 hybrid_clock.cc:648] HybridClock initialized: now 1761335273866115 us; error 56 us; skew 500 ppm
I20251024 19:47:53.866631 1289 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251024 19:47:53.870235 1289 webserver.cc:492] Webserver started at http://127.0.252.193:45257/ using document root <none> and password file <none>
I20251024 19:47:53.871439 1289 fs_manager.cc:362] Metadata directory not provided
I20251024 19:47:53.871613 1289 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 19:47:53.872246 1289 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 19:47:53.875895 1289 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/data/instance:
uuid: "43bec2f03fc849099e254f6335e9d532"
format_stamp: "Formatted at 2025-10-24 19:47:53 on dist-test-slave-53sn"
I20251024 19:47:53.877036 1289 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/instance:
uuid: "43bec2f03fc849099e254f6335e9d532"
format_stamp: "Formatted at 2025-10-24 19:47:53 on dist-test-slave-53sn"
I20251024 19:47:53.882580 1302 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc8f71a5b70749a4a3e3a599bcb3d22c)
I20251024 19:47:53.883015 1302 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:47:53.884398 1302 leader_election.cc:290] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82e82847f8904b59b03f446b7fce6233 (127.0.252.253:41375), dc8f71a5b70749a4a3e3a599bcb3d22c (127.0.252.252:36627)
I20251024 19:47:53.884773 1289 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20251024 19:47:53.885077 1129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef8906492b1a4368936fac95c11e1f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82e82847f8904b59b03f446b7fce6233" is_pre_election: true
I20251024 19:47:53.885208 1202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef8906492b1a4368936fac95c11e1f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" is_pre_election: true
I20251024 19:47:53.886081 1037 leader_election.cc:304] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef8906492b1a4368936fac95c11e1f73; no voters: 82e82847f8904b59b03f446b7fce6233, dc8f71a5b70749a4a3e3a599bcb3d22c
I20251024 19:47:53.886435 1302 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:47:53.889882 1304 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 19:47:53.891556 1289 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251024 19:47:53.891706 1289 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal
uuid: "43bec2f03fc849099e254f6335e9d532"
format_stamp: "Formatted at 2025-10-24 19:47:53 on dist-test-slave-53sn"
I20251024 19:47:53.891899 1289 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 19:47:53.924333 1289 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 19:47:53.925194 1289 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 19:47:53.925426 1289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 19:47:53.926414 1289 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 19:47:53.928054 1289 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 19:47:53.928143 1289 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 19:47:53.928220 1289 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 19:47:53.928273 1289 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 19:47:53.971952 1289 rpc_server.cc:307] RPC server started. Bound to: 127.0.252.193:36103
I20251024 19:47:53.972023 1416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.252.193:36103 every 8 connection(s)
I20251024 19:47:53.973604 1289 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/data/info.pb
I20251024 19:47:53.975107 1011 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskc9NInd/build/asan/bin/kudu as pid 1289
I20251024 19:47:53.975370 1011 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/instance
I20251024 19:47:53.981257 1011 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskc9NInd/build/asan/bin/kudu
/tmp/dist-test-taskc9NInd/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.252.194:0
--local_ip_for_outbound_sockets=127.0.252.194
--webserver_interface=127.0.252.194
--webserver_port=0
--tserver_master_addrs=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--builtin_ntp_servers=127.0.252.212:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251024 19:47:53.997830 1417 heartbeater.cc:344] Connected to a master server at 127.0.252.252:36627
I20251024 19:47:53.998436 1417 heartbeater.cc:461] Registering TS with master...
I20251024 19:47:54.000111 1418 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:47:54.000385 1418 heartbeater.cc:461] Registering TS with master...
I20251024 19:47:54.001075 1417 heartbeater.cc:507] Master 127.0.252.252:36627 requested a full tablet report, sending...
I20251024 19:47:54.002233 1418 heartbeater.cc:507] Master 127.0.252.254:36505 requested a full tablet report, sending...
I20251024 19:47:54.004241 1049 ts_manager.cc:194] Registered new tserver with Master: 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:47:54.004433 1192 ts_manager.cc:194] Registered new tserver with Master: 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:47:54.005786 1419 heartbeater.cc:344] Connected to a master server at 127.0.252.253:41375
I20251024 19:47:54.005995 1419 heartbeater.cc:461] Registering TS with master...
I20251024 19:47:54.006426 1419 heartbeater.cc:507] Master 127.0.252.253:41375 requested a full tablet report, sending...
I20251024 19:47:54.006901 1192 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.252.193:37629
I20251024 19:47:54.007947 1120 ts_manager.cc:194] Registered new tserver with Master: 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:47:54.257215 1265 catalog_manager.cc:1269] Loaded cluster ID: 57054fc705ad43a6b8ba5fe9c17ff20a
I20251024 19:47:54.257380 1265 catalog_manager.cc:1562] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: loading cluster ID for follower catalog manager: success
I20251024 19:47:54.260222 1265 catalog_manager.cc:1584] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: acquiring CA information for follower catalog manager: success
I20251024 19:47:54.262060 1265 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251024 19:47:54.295672 1428 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc8f71a5b70749a4a3e3a599bcb3d22c)
I20251024 19:47:54.296064 1428 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:47:54.297914 1428 leader_election.cc:290] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef8906492b1a4368936fac95c11e1f73 (127.0.252.254:36505), dc8f71a5b70749a4a3e3a599bcb3d22c (127.0.252.252:36627)
I20251024 19:47:54.298362 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e82847f8904b59b03f446b7fce6233" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef8906492b1a4368936fac95c11e1f73" is_pre_election: true
I20251024 19:47:54.298668 1202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e82847f8904b59b03f446b7fce6233" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" is_pre_election: true
I20251024 19:47:54.299386 1108 leader_election.cc:304] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82e82847f8904b59b03f446b7fce6233; no voters: dc8f71a5b70749a4a3e3a599bcb3d22c, ef8906492b1a4368936fac95c11e1f73
I20251024 19:47:54.299813 1428 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:47:54.370954 1288 catalog_manager.cc:1269] Loaded cluster ID: 57054fc705ad43a6b8ba5fe9c17ff20a
I20251024 19:47:54.371145 1288 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233: loading cluster ID for follower catalog manager: success
I20251024 19:47:54.375710 1288 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233: acquiring CA information for follower catalog manager: success
I20251024 19:47:54.378528 1288 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251024 19:47:54.391525 1427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 19:47:54.391790 1427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 19:47:54.391932 1427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 19:47:54.400879 1427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 19:47:54.401064 1427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.252.194
I20251024 19:47:54.411736 1427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.252.212:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.252.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.252.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.252.194
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 24 Oct 2025 19:43:25 UTC on 24a791456cd2
build id 8700
ASAN enabled
I20251024 19:47:54.413317 1427 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 19:47:54.415167 1427 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 19:47:54.424650 1435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:47:54.425468 1434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:47:54.427170 1437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 19:47:54.427435 1427 server_base.cc:1047] running on GCE node
I20251024 19:47:54.428143 1427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 19:47:54.429455 1427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 19:47:54.430696 1427 hybrid_clock.cc:648] HybridClock initialized: now 1761335274430640 us; error 61 us; skew 500 ppm
I20251024 19:47:54.431066 1427 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251024 19:47:54.433768 1427 webserver.cc:492] Webserver started at http://127.0.252.194:33509/ using document root <none> and password file <none>
I20251024 19:47:54.434744 1427 fs_manager.cc:362] Metadata directory not provided
I20251024 19:47:54.434948 1427 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 19:47:54.435586 1427 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 19:47:54.439617 1427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/data/instance:
uuid: "10a9536207e343f188e0ef709081d04e"
format_stamp: "Formatted at 2025-10-24 19:47:54 on dist-test-slave-53sn"
I20251024 19:47:54.440959 1427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/instance:
uuid: "10a9536207e343f188e0ef709081d04e"
format_stamp: "Formatted at 2025-10-24 19:47:54 on dist-test-slave-53sn"
I20251024 19:47:54.447764 1427 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20251024 19:47:54.452064 1443 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 19:47:54.453518 1427 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251024 19:47:54.453702 1427 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal
uuid: "10a9536207e343f188e0ef709081d04e"
format_stamp: "Formatted at 2025-10-24 19:47:54 on dist-test-slave-53sn"
I20251024 19:47:54.453919 1427 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 19:47:54.496601 1427 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 19:47:54.498258 1427 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 19:47:54.498673 1427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 19:47:54.499982 1427 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 19:47:54.502349 1427 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 19:47:54.502475 1427 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 19:47:54.502604 1427 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 19:47:54.502681 1427 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 19:47:54.510576 1417 heartbeater.cc:499] Master 127.0.252.252:36627 was elected leader, sending a full tablet report...
I20251024 19:47:54.572058 1427 rpc_server.cc:307] RPC server started. Bound to: 127.0.252.194:34045
I20251024 19:47:54.572151 1555 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.252.194:34045 every 8 connection(s)
I20251024 19:47:54.574397 1427 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/data/info.pb
I20251024 19:47:54.581321 1011 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskc9NInd/build/asan/bin/kudu as pid 1427
I20251024 19:47:54.581598 1011 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/instance
I20251024 19:47:54.587635 1011 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskc9NInd/build/asan/bin/kudu
/tmp/dist-test-taskc9NInd/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.252.195:0
--local_ip_for_outbound_sockets=127.0.252.195
--webserver_interface=127.0.252.195
--webserver_port=0
--tserver_master_addrs=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--builtin_ntp_servers=127.0.252.212:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251024 19:47:54.610457 1558 heartbeater.cc:344] Connected to a master server at 127.0.252.253:41375
I20251024 19:47:54.610929 1558 heartbeater.cc:461] Registering TS with master...
I20251024 19:47:54.612169 1558 heartbeater.cc:507] Master 127.0.252.253:41375 requested a full tablet report, sending...
I20251024 19:47:54.614485 1120 ts_manager.cc:194] Registered new tserver with Master: 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:47:54.618023 1557 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:47:54.618161 1557 heartbeater.cc:461] Registering TS with master...
I20251024 19:47:54.618530 1556 heartbeater.cc:344] Connected to a master server at 127.0.252.252:36627
I20251024 19:47:54.618723 1556 heartbeater.cc:461] Registering TS with master...
I20251024 19:47:54.618727 1557 heartbeater.cc:507] Master 127.0.252.254:36505 requested a full tablet report, sending...
I20251024 19:47:54.619313 1556 heartbeater.cc:507] Master 127.0.252.252:36627 requested a full tablet report, sending...
I20251024 19:47:54.619979 1049 ts_manager.cc:194] Registered new tserver with Master: 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:47:54.620292 1192 ts_manager.cc:194] Registered new tserver with Master: 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:47:54.621517 1192 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.252.194:36397
I20251024 19:47:54.834684 1567 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc8f71a5b70749a4a3e3a599bcb3d22c)
I20251024 19:47:54.835057 1567 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:47:54.836643 1567 leader_election.cc:290] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef8906492b1a4368936fac95c11e1f73 (127.0.252.254:36505), dc8f71a5b70749a4a3e3a599bcb3d22c (127.0.252.252:36627)
I20251024 19:47:54.836862 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e82847f8904b59b03f446b7fce6233" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef8906492b1a4368936fac95c11e1f73" is_pre_election: true
I20251024 19:47:54.837669 1202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e82847f8904b59b03f446b7fce6233" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" is_pre_election: true
I20251024 19:47:54.838354 1108 leader_election.cc:304] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82e82847f8904b59b03f446b7fce6233; no voters: dc8f71a5b70749a4a3e3a599bcb3d22c, ef8906492b1a4368936fac95c11e1f73
I20251024 19:47:54.838706 1567 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:47:54.891301 1568 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc8f71a5b70749a4a3e3a599bcb3d22c)
I20251024 19:47:54.891486 1568 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:47:54.892174 1568 leader_election.cc:290] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82e82847f8904b59b03f446b7fce6233 (127.0.252.253:41375), dc8f71a5b70749a4a3e3a599bcb3d22c (127.0.252.252:36627)
I20251024 19:47:54.892727 1202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef8906492b1a4368936fac95c11e1f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" is_pre_election: true
I20251024 19:47:54.892843 1129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef8906492b1a4368936fac95c11e1f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82e82847f8904b59b03f446b7fce6233" is_pre_election: true
I20251024 19:47:54.893425 1037 leader_election.cc:304] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef8906492b1a4368936fac95c11e1f73; no voters: 82e82847f8904b59b03f446b7fce6233, dc8f71a5b70749a4a3e3a599bcb3d22c
I20251024 19:47:54.893720 1568 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251024 19:47:55.005765 1566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 19:47:55.006117 1566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 19:47:55.006260 1566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 19:47:55.015861 1566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 19:47:55.016098 1566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.252.195
I20251024 19:47:55.027576 1566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.252.212:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.252.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.252.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.252.195
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 24 Oct 2025 19:43:25 UTC on 24a791456cd2
build id 8700
ASAN enabled
I20251024 19:47:55.029237 1566 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 19:47:55.031121 1566 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 19:47:55.041157 1574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:47:55.042156 1573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:47:55.044652 1576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 19:47:55.044662 1566 server_base.cc:1047] running on GCE node
I20251024 19:47:55.045464 1566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 19:47:55.046897 1566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 19:47:55.048242 1566 hybrid_clock.cc:648] HybridClock initialized: now 1761335275048128 us; error 58 us; skew 500 ppm
I20251024 19:47:55.048650 1566 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251024 19:47:55.051868 1566 webserver.cc:492] Webserver started at http://127.0.252.195:46079/ using document root <none> and password file <none>
I20251024 19:47:55.052978 1566 fs_manager.cc:362] Metadata directory not provided
I20251024 19:47:55.053161 1566 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 19:47:55.053648 1566 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 19:47:55.056037 1566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/data/instance:
uuid: "af3952de050140d383f6b0de7666fa97"
format_stamp: "Formatted at 2025-10-24 19:47:55 on dist-test-slave-53sn"
I20251024 19:47:55.056836 1566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/instance:
uuid: "af3952de050140d383f6b0de7666fa97"
format_stamp: "Formatted at 2025-10-24 19:47:55 on dist-test-slave-53sn"
I20251024 19:47:55.062397 1566 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20251024 19:47:55.065701 1582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 19:47:55.066951 1566 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251024 19:47:55.067099 1566 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal
uuid: "af3952de050140d383f6b0de7666fa97"
format_stamp: "Formatted at 2025-10-24 19:47:55 on dist-test-slave-53sn"
I20251024 19:47:55.067283 1566 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 19:47:55.093341 1566 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 19:47:55.094331 1566 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 19:47:55.094604 1566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 19:47:55.095706 1566 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 19:47:55.097910 1566 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 19:47:55.098016 1566 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 19:47:55.098124 1566 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 19:47:55.098196 1566 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 19:47:55.124963 1556 heartbeater.cc:499] Master 127.0.252.252:36627 was elected leader, sending a full tablet report...
I20251024 19:47:55.151129 1566 rpc_server.cc:307] RPC server started. Bound to: 127.0.252.195:34119
I20251024 19:47:55.151199 1694 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.252.195:34119 every 8 connection(s)
I20251024 19:47:55.152525 1566 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/data/info.pb
I20251024 19:47:55.155133 1011 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskc9NInd/build/asan/bin/kudu as pid 1566
I20251024 19:47:55.155303 1011 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/instance
I20251024 19:47:55.172199 1695 heartbeater.cc:344] Connected to a master server at 127.0.252.252:36627
I20251024 19:47:55.172720 1695 heartbeater.cc:461] Registering TS with master...
I20251024 19:47:55.173710 1695 heartbeater.cc:507] Master 127.0.252.252:36627 requested a full tablet report, sending...
I20251024 19:47:55.175202 1192 ts_manager.cc:194] Registered new tserver with Master: af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:55.176016 1192 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.252.195:44995
I20251024 19:47:55.184717 1696 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:47:55.184921 1696 heartbeater.cc:461] Registering TS with master...
I20251024 19:47:55.185457 1696 heartbeater.cc:507] Master 127.0.252.254:36505 requested a full tablet report, sending...
I20251024 19:47:55.187053 1049 ts_manager.cc:194] Registered new tserver with Master: af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:55.188966 1697 heartbeater.cc:344] Connected to a master server at 127.0.252.253:41375
I20251024 19:47:55.189124 1697 heartbeater.cc:461] Registering TS with master...
I20251024 19:47:55.189795 1697 heartbeater.cc:507] Master 127.0.252.253:41375 requested a full tablet report, sending...
I20251024 19:47:55.191111 1121 ts_manager.cc:194] Registered new tserver with Master: af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:55.192058 1011 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251024 19:47:55.232098 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f76d56ad69134a28ab8f78f3e3298850"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.232518 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_107c52b6479b4df2b51043399a1b4a16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.234814 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_707688130a0f4a09ab320bf53d024121"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.235023 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_107c52b6479b4df2b51043399a1b4a16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:55.235025 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f76d56ad69134a28ab8f78f3e3298850 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:55.238947 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_707688130a0f4a09ab320bf53d024121 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.235543 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a462baf87c6f45dbb111d7c6c1166ab7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.235999 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_41223bd224814375b0006f6a99006972"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.238734 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f79367cb60dc48e491fe71b40407cfdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.244266 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a72ebe2501594ccba9c78288445ecbb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.244725 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d023e116571d4cc4b3790025e8e8b212"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.244382 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a462baf87c6f45dbb111d7c6c1166ab7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.238797 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4dc587d5bfdd4c78ab7b4118083868cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.246829 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_97681ead2b1e467ca67bc980c8f13633"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.248737 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f79367cb60dc48e491fe71b40407cfdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:55.248847 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dc587d5bfdd4c78ab7b4118083868cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:55.245621 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d023e116571d4cc4b3790025e8e8b212 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:55.245332 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a72ebe2501594ccba9c78288445ecbb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:55.253904 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41223bd224814375b0006f6a99006972 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:55.255873 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97681ead2b1e467ca67bc980c8f13633 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.308423 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f76d56ad69134a28ab8f78f3e3298850" } new_table_name: "default.Table_156e684f2784478ca8353d00960aa194" modify_external_catalogs: true
I20251024 19:47:55.330008 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_707688130a0f4a09ab320bf53d024121" } new_table_name: "default.Table_835ab1cc0a82456f8282bce320dca4f8" modify_external_catalogs: true
I20251024 19:47:55.330698 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4dc587d5bfdd4c78ab7b4118083868cc" } new_table_name: "default.Table_47f1b33ad99f4acfa21028882921876f" modify_external_catalogs: true
I20251024 19:47:55.331480 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f79367cb60dc48e491fe71b40407cfdb" } new_table_name: "default.Table_034ccb00682e4e58a41d79be2c97b364" modify_external_catalogs: true
I20251024 19:47:55.334556 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a72ebe2501594ccba9c78288445ecbb3" } modify_external_catalogs: true
I20251024 19:47:55.334800 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a72ebe2501594ccba9c78288445ecbb3" } modify_external_catalogs: true
I20251024 19:47:55.336745 1011 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskc9NInd/build/asan/bin/kudu pid 1019
I20251024 19:47:55.337061 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a462baf87c6f45dbb111d7c6c1166ab7" } new_table_name: "default.Table_ded6a4ffec4f48b3952d51d5b43eae12" modify_external_catalogs: true
I20251024 19:47:55.338743 1080 generic_service.cc:196] Checking for leaks (request via RPC)
I20251024 19:47:55.340533 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_107c52b6479b4df2b51043399a1b4a16" } modify_external_catalogs: true
I20251024 19:47:55.340905 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_107c52b6479b4df2b51043399a1b4a16" } modify_external_catalogs: true
I20251024 19:47:55.341835 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_41223bd224814375b0006f6a99006972" } modify_external_catalogs: true
I20251024 19:47:55.342100 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_41223bd224814375b0006f6a99006972" } modify_external_catalogs: true
I20251024 19:47:55.343557 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d023e116571d4cc4b3790025e8e8b212" } modify_external_catalogs: true
I20251024 19:47:55.344007 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d023e116571d4cc4b3790025e8e8b212" } modify_external_catalogs: true
I20251024 19:47:55.345970 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_97681ead2b1e467ca67bc980c8f13633" } modify_external_catalogs: true
I20251024 19:47:55.346208 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_97681ead2b1e467ca67bc980c8f13633" } modify_external_catalogs: true
W20251024 19:47:55.364517 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f34c1080c5554c66b91c99ea294632b4 has no consensus state
W20251024 19:47:55.364535 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f9a6a1dc46744a385cdb484524c5c59 has no consensus state
W20251024 19:47:55.370604 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad673e92efc94d33ad07a02b6464e724 has no consensus state
W20251024 19:47:55.371125 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5386938a8faf40e0ba1c7838bfbc35bc has no consensus state
I20251024 19:47:55.371577 1189 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 70c98592062449a78d9c8c9e0d1b975f
W20251024 19:47:55.372028 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2218f5fe8f77443992a9b83c694e2d20 has no consensus state
I20251024 19:47:55.392637 1185 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 41c78287464047a98f7079ca375976cc
I20251024 19:47:55.393043 1190 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 91948489b6c94597a1a0da586554d0d1
I20251024 19:47:55.393361 1186 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet a8ce58e1e2da40f59339b63bb9c6666a
I20251024 19:47:55.395709 1183 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 7c95af34cd09462dbf6a663b018200d0
W20251024 19:47:55.397701 1179 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f34c1080c5554c66b91c99ea294632b4 (table default.Table_835ab1cc0a82456f8282bce320dca4f8 [id=91d53300d972437d9efba6385dbef72c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f34c1080c5554c66b91c99ea294632b4 has no consensus state
I20251024 19:47:55.402056 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 7f9a6a1dc46744a385cdb484524c5c59 (DEFAULT_TABLE table=default.Table_156e684f2784478ca8353d00960aa194 [id=2fd5c541b06a46469e65ed0942d2c394]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.403952 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f9a6a1dc46744a385cdb484524c5c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.413944 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 5386938a8faf40e0ba1c7838bfbc35bc (DEFAULT_TABLE table=default.Table_034ccb00682e4e58a41d79be2c97b364 [id=aadca5a1b6d4494aa37f8a8cc76ad12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.414009 1350 tablet_service.cc:1505] Processing CreateTablet for tablet f34c1080c5554c66b91c99ea294632b4 (DEFAULT_TABLE table=default.Table_835ab1cc0a82456f8282bce320dca4f8 [id=91d53300d972437d9efba6385dbef72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.414572 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5386938a8faf40e0ba1c7838bfbc35bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.414764 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 2218f5fe8f77443992a9b83c694e2d20 (DEFAULT_TABLE table=default.Table_ded6a4ffec4f48b3952d51d5b43eae12 [id=10bb93537e8344b2bf08a46e2f1109a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.415498 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2218f5fe8f77443992a9b83c694e2d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.418284 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f34c1080c5554c66b91c99ea294632b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.421408 1349 tablet_service.cc:1505] Processing CreateTablet for tablet ad673e92efc94d33ad07a02b6464e724 (DEFAULT_TABLE table=default.Table_47f1b33ad99f4acfa21028882921876f [id=750d9803961d4003b052aa142e097a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.422452 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad673e92efc94d33ad07a02b6464e724. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.423730 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 7f9a6a1dc46744a385cdb484524c5c59 (DEFAULT_TABLE table=default.Table_156e684f2784478ca8353d00960aa194 [id=2fd5c541b06a46469e65ed0942d2c394]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.424050 1628 tablet_service.cc:1505] Processing CreateTablet for tablet f34c1080c5554c66b91c99ea294632b4 (DEFAULT_TABLE table=default.Table_835ab1cc0a82456f8282bce320dca4f8 [id=91d53300d972437d9efba6385dbef72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.425470 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f9a6a1dc46744a385cdb484524c5c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.425729 1627 tablet_service.cc:1505] Processing CreateTablet for tablet ad673e92efc94d33ad07a02b6464e724 (DEFAULT_TABLE table=default.Table_47f1b33ad99f4acfa21028882921876f [id=750d9803961d4003b052aa142e097a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.426312 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 5386938a8faf40e0ba1c7838bfbc35bc (DEFAULT_TABLE table=default.Table_034ccb00682e4e58a41d79be2c97b364 [id=aadca5a1b6d4494aa37f8a8cc76ad12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.427177 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5386938a8faf40e0ba1c7838bfbc35bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.427366 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad673e92efc94d33ad07a02b6464e724. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.428515 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 7f9a6a1dc46744a385cdb484524c5c59 (DEFAULT_TABLE table=default.Table_156e684f2784478ca8353d00960aa194 [id=2fd5c541b06a46469e65ed0942d2c394]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.429623 1626 tablet_service.cc:1505] Processing CreateTablet for tablet 2218f5fe8f77443992a9b83c694e2d20 (DEFAULT_TABLE table=default.Table_ded6a4ffec4f48b3952d51d5b43eae12 [id=10bb93537e8344b2bf08a46e2f1109a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.429955 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f9a6a1dc46744a385cdb484524c5c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.430353 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2218f5fe8f77443992a9b83c694e2d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.432137 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f34c1080c5554c66b91c99ea294632b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.457461 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 5386938a8faf40e0ba1c7838bfbc35bc (DEFAULT_TABLE table=default.Table_034ccb00682e4e58a41d79be2c97b364 [id=aadca5a1b6d4494aa37f8a8cc76ad12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.458314 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5386938a8faf40e0ba1c7838bfbc35bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.469925 1489 tablet_service.cc:1505] Processing CreateTablet for tablet ad673e92efc94d33ad07a02b6464e724 (DEFAULT_TABLE table=default.Table_47f1b33ad99f4acfa21028882921876f [id=750d9803961d4003b052aa142e097a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.470479 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad673e92efc94d33ad07a02b6464e724. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.473845 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 2218f5fe8f77443992a9b83c694e2d20 (DEFAULT_TABLE table=default.Table_ded6a4ffec4f48b3952d51d5b43eae12 [id=10bb93537e8344b2bf08a46e2f1109a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.474324 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2218f5fe8f77443992a9b83c694e2d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.477059 1768 tablet_bootstrap.cc:492] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:55.481564 1768 tablet_bootstrap.cc:654] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.483062 1768 log.cc:826] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532: Log is configured to *not* fsync() on all Append() calls
I20251024 19:47:55.483945 1767 tablet_bootstrap.cc:492] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:55.492156 1766 tablet_bootstrap.cc:492] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:55.493405 1768 tablet_bootstrap.cc:492] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:55.493556 1490 tablet_service.cc:1505] Processing CreateTablet for tablet f34c1080c5554c66b91c99ea294632b4 (DEFAULT_TABLE table=default.Table_835ab1cc0a82456f8282bce320dca4f8 [id=91d53300d972437d9efba6385dbef72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.493759 1768 ts_tablet_manager.cc:1403] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.003s
I20251024 19:47:55.494112 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f34c1080c5554c66b91c99ea294632b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.495155 1767 tablet_bootstrap.cc:654] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.499192 1767 log.cc:826] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97: Log is configured to *not* fsync() on all Append() calls
I20251024 19:47:55.501371 1766 tablet_bootstrap.cc:654] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.501827 1768 raft_consensus.cc:359] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:55.502401 1768 raft_consensus.cc:385] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.502570 1768 raft_consensus.cc:740] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.502827 1766 log.cc:826] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e: Log is configured to *not* fsync() on all Append() calls
I20251024 19:47:55.503451 1768 consensus_queue.cc:260] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:55.508129 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6335db260a264179aef9dbf353382a6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.508873 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6335db260a264179aef9dbf353382a6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.509959 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet ad673e92efc94d33ad07a02b6464e724 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:55.511314 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet 2218f5fe8f77443992a9b83c694e2d20 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:55.512379 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet f34c1080c5554c66b91c99ea294632b4 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:55.516461 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet f34c1080c5554c66b91c99ea294632b4 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:55.516695 1768 ts_tablet_manager.cc:1434] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.023s user 0.009s sys 0.007s
I20251024 19:47:55.516800 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_41507c038ecb4e8ba8aadd563d12b6bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.517197 1768 tablet_bootstrap.cc:492] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:47:55.517328 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41507c038ecb4e8ba8aadd563d12b6bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.518514 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c4ffe38e352744a3891e5b1435036e9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.518961 1768 tablet_bootstrap.cc:654] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:47:55.518972 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4ffe38e352744a3891e5b1435036e9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.523411 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2218f5fe8f77443992a9b83c694e2d20
I20251024 19:47:55.524041 1190 catalog_manager.cc:6719] ReplaceTablet: tablet 2218f5fe8f77443992a9b83c694e2d20 deleted and replaced by tablet 9af5d5bd80a14f2ebdc7a595b17416e6
I20251024 19:47:55.527055 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 2218f5fe8f77443992a9b83c694e2d20 with delete_type TABLET_DATA_DELETED (replaced by tablet 9af5d5bd80a14f2ebdc7a595b17416e6) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:55.527316 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 2218f5fe8f77443992a9b83c694e2d20 with delete_type TABLET_DATA_DELETED (replaced by tablet 9af5d5bd80a14f2ebdc7a595b17416e6) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:55.528172 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ad673e92efc94d33ad07a02b6464e724
I20251024 19:47:55.528638 1186 catalog_manager.cc:6719] ReplaceTablet: tablet ad673e92efc94d33ad07a02b6464e724 deleted and replaced by tablet 44d777147a60462e86b3abf18a40f2c9
I20251024 19:47:55.529381 1627 tablet_service.cc:1552] Processing DeleteTablet for tablet 2218f5fe8f77443992a9b83c694e2d20 with delete_type TABLET_DATA_DELETED (replaced by tablet 9af5d5bd80a14f2ebdc7a595b17416e6) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:55.530485 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_36406152efad4ca4b1a106ed81d39fbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.530545 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e7a8fc2ba8e0481cb7479d33f3ecbdf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.531092 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7a8fc2ba8e0481cb7479d33f3ecbdf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:55.531127 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36406152efad4ca4b1a106ed81d39fbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.531176 1767 tablet_bootstrap.cc:492] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:55.531497 1767 ts_tablet_manager.cc:1403] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.000s
I20251024 19:47:55.533056 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet ad673e92efc94d33ad07a02b6464e724 with delete_type TABLET_DATA_DELETED (replaced by tablet 44d777147a60462e86b3abf18a40f2c9) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:55.533114 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet ad673e92efc94d33ad07a02b6464e724 with delete_type TABLET_DATA_DELETED (replaced by tablet 44d777147a60462e86b3abf18a40f2c9) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:55.535336 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet ad673e92efc94d33ad07a02b6464e724 with delete_type TABLET_DATA_DELETED (replaced by tablet 44d777147a60462e86b3abf18a40f2c9) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:55.536933 1766 tablet_bootstrap.cc:492] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:55.539184 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_907792b4009a4fbfa4b4cca9a93ec64d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.539746 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_907792b4009a4fbfa4b4cca9a93ec64d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.540875 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1e0a5d980ca648e28cee93bc3f5607d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.541347 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e0a5d980ca648e28cee93bc3f5607d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.542949 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ae01a61e72ea4e9592e345869e67e586"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.543370 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae01a61e72ea4e9592e345869e67e586 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.546025 1766 ts_tablet_manager.cc:1403] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.010s
I20251024 19:47:55.547962 1772 raft_consensus.cc:493] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:55.548324 1772 raft_consensus.cc:515] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:55.550733 1772 leader_election.cc:290] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:47:55.554123 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f34c1080c5554c66b91c99ea294632b4
I20251024 19:47:55.554726 1185 catalog_manager.cc:6719] ReplaceTablet: tablet f34c1080c5554c66b91c99ea294632b4 deleted and replaced by tablet 20f32e34ce51446d8cc7d4938017e60e
I20251024 19:47:55.555289 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet f34c1080c5554c66b91c99ea294632b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 20f32e34ce51446d8cc7d4938017e60e) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:55.555245 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_46d6218260e8421b9093ea1630d5c8de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.555922 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46d6218260e8421b9093ea1630d5c8de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.558866 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2759324fce09406ab532d22228541f6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.559633 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2759324fce09406ab532d22228541f6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:55.565680 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2218f5fe8f77443992a9b83c694e2d20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2218f5fe8f77443992a9b83c694e2d20 already in progress: creating tablet
I20251024 19:47:55.567019 1189 master_service.cc:954] ReplaceTablet: received request to replace tablet 7f9a6a1dc46744a385cdb484524c5c59 from {username='slave'} at 127.0.0.1:51416
W20251024 19:47:55.567821 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet ad673e92efc94d33ad07a02b6464e724 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad673e92efc94d33ad07a02b6464e724 already in progress: creating tablet
W20251024 19:47:55.568526 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f34c1080c5554c66b91c99ea294632b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f34c1080c5554c66b91c99ea294632b4 already in progress: creating tablet
I20251024 19:47:55.580865 1627 tablet_service.cc:1505] Processing CreateTablet for tablet af2cc8c940b94b0ab10bfc98c8618e6f (DEFAULT_TABLE table=default.Table_6335db260a264179aef9dbf353382a6d [id=2c92cb09b287410c90b811e4b88c2260]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.581445 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af2cc8c940b94b0ab10bfc98c8618e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.583707 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet f34c1080c5554c66b91c99ea294632b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 20f32e34ce51446d8cc7d4938017e60e) from {username='slave'} at 127.0.0.1:46462
W20251024 19:47:55.585875 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2218f5fe8f77443992a9b83c694e2d20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2218f5fe8f77443992a9b83c694e2d20 already in progress: creating tablet
I20251024 19:47:55.586089 1490 tablet_service.cc:1505] Processing CreateTablet for tablet af2cc8c940b94b0ab10bfc98c8618e6f (DEFAULT_TABLE table=default.Table_6335db260a264179aef9dbf353382a6d [id=2c92cb09b287410c90b811e4b88c2260]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.586529 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af2cc8c940b94b0ab10bfc98c8618e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.587072 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_47f1b33ad99f4acfa21028882921876f" } new_table_name: "default.Table_7ddfa79850884c62900d2e92bc798c64" modify_external_catalogs: true
I20251024 19:47:55.588773 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ded6a4ffec4f48b3952d51d5b43eae12" } new_table_name: "default.Table_73de54ad8b6b436097fe701dfd52d703" modify_external_catalogs: true
W20251024 19:47:55.590117 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2218f5fe8f77443992a9b83c694e2d20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2218f5fe8f77443992a9b83c694e2d20 already in progress: creating tablet
W20251024 19:47:55.590443 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet ad673e92efc94d33ad07a02b6464e724 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad673e92efc94d33ad07a02b6464e724 already in progress: creating tablet
W20251024 19:47:55.590880 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet f34c1080c5554c66b91c99ea294632b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f34c1080c5554c66b91c99ea294632b4 already in progress: creating tablet
W20251024 19:47:55.595643 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44d777147a60462e86b3abf18a40f2c9 has no consensus state
W20251024 19:47:55.595767 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9af5d5bd80a14f2ebdc7a595b17416e6 has no consensus state
I20251024 19:47:55.599493 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_156e684f2784478ca8353d00960aa194" } new_table_name: "default.Table_7fb527771f604f21beccb4abca21a38f" modify_external_catalogs: true
I20251024 19:47:55.602762 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_907792b4009a4fbfa4b4cca9a93ec64d" } new_table_name: "default.Table_a66b74f191194717bd7ab791fbeaa0f0" modify_external_catalogs: true
I20251024 19:47:55.605881 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1e0a5d980ca648e28cee93bc3f5607d7" } modify_external_catalogs: true
I20251024 19:47:55.606230 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1e0a5d980ca648e28cee93bc3f5607d7" } modify_external_catalogs: true
W20251024 19:47:55.611300 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e31eef151884385854096ff71484a9c has no consensus state
I20251024 19:47:55.611218 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_034ccb00682e4e58a41d79be2c97b364" } new_table_name: "default.Table_8b6c3241e78148feadc4a16fc4c00582" modify_external_catalogs: true
I20251024 19:47:55.616094 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ae01a61e72ea4e9592e345869e67e586" } modify_external_catalogs: true
I20251024 19:47:55.616559 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ae01a61e72ea4e9592e345869e67e586" } modify_external_catalogs: true
I20251024 19:47:55.618100 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2759324fce09406ab532d22228541f6e" } modify_external_catalogs: true
I20251024 19:47:55.618341 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2759324fce09406ab532d22228541f6e" } modify_external_catalogs: true
I20251024 19:47:55.621559 1185 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet d65216c0414f4a589fbfa436f8ad5095
W20251024 19:47:55.626086 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5386938a8faf40e0ba1c7838bfbc35bc consensus state has no leader
I20251024 19:47:55.627066 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_835ab1cc0a82456f8282bce320dca4f8" } modify_external_catalogs: true
I20251024 19:47:55.627316 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_835ab1cc0a82456f8282bce320dca4f8" } modify_external_catalogs: true
I20251024 19:47:55.628330 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_36406152efad4ca4b1a106ed81d39fbe" } modify_external_catalogs: true
I20251024 19:47:55.628513 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_36406152efad4ca4b1a106ed81d39fbe" } modify_external_catalogs: true
I20251024 19:47:55.629001 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet f34c1080c5554c66b91c99ea294632b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 20f32e34ce51446d8cc7d4938017e60e) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:55.629897 1350 tablet_service.cc:1505] Processing CreateTablet for tablet af2cc8c940b94b0ab10bfc98c8618e6f (DEFAULT_TABLE table=default.Table_6335db260a264179aef9dbf353382a6d [id=2c92cb09b287410c90b811e4b88c2260]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.630288 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af2cc8c940b94b0ab10bfc98c8618e6f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:55.633653 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet ad673e92efc94d33ad07a02b6464e724 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad673e92efc94d33ad07a02b6464e724 already in progress: creating tablet
I20251024 19:47:55.633957 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f9a6a1dc46744a385cdb484524c5c59" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:47:55.635432 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet f34c1080c5554c66b91c99ea294632b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f34c1080c5554c66b91c99ea294632b4 already in progress: creating tablet
W20251024 19:47:55.638595 1305 leader_election.cc:343] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:55.639181 1768 tablet_bootstrap.cc:492] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:55.639413 1768 ts_tablet_manager.cc:1403] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.122s user 0.026s sys 0.026s
I20251024 19:47:55.640925 1768 raft_consensus.cc:359] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.641108 1768 raft_consensus.cc:385] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.641189 1768 raft_consensus.cc:740] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.641506 1768 consensus_queue.cc:260] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.604079 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f9a6a1dc46744a385cdb484524c5c59" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:47:55.643738 1308 leader_election.cc:343] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:55.643880 1308 leader_election.cc:304] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:55.644721 1772 raft_consensus.cc:2749] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:55.645774 1768 ts_tablet_manager.cc:1434] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251024 19:47:55.646697 1011 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskc9NInd/build/asan/bin/kudu with pid 1019
I20251024 19:47:55.647764 1768 tablet_bootstrap.cc:492] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:55.649046 1766 raft_consensus.cc:359] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:55.649561 1766 raft_consensus.cc:385] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.649755 1766 raft_consensus.cc:740] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.650727 1768 tablet_bootstrap.cc:654] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.650591 1766 consensus_queue.cc:260] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:55.657274 1627 tablet_service.cc:1505] Processing CreateTablet for tablet 0df102c2be684823a8eb3b9c5d40d744 (DEFAULT_TABLE table=default.Table_ae01a61e72ea4e9592e345869e67e586 [id=41f77affc9b0465d8159bbbc2d16646c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.657815 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0df102c2be684823a8eb3b9c5d40d744. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.658272 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 0df102c2be684823a8eb3b9c5d40d744 (DEFAULT_TABLE table=default.Table_ae01a61e72ea4e9592e345869e67e586 [id=41f77affc9b0465d8159bbbc2d16646c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.658697 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0df102c2be684823a8eb3b9c5d40d744. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.659024 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 7e31eef151884385854096ff71484a9c (DEFAULT_TABLE table=default.Table_a66b74f191194717bd7ab791fbeaa0f0 [id=4e6695ccb7d342a6985b2a10a6bdfaac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.659387 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e31eef151884385854096ff71484a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.663269 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 26d6f1105b324afba36081c760930ef0
I20251024 19:47:55.664345 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 26d6f1105b324afba36081c760930ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:55.665120 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 7e31eef151884385854096ff71484a9c (DEFAULT_TABLE table=default.Table_a66b74f191194717bd7ab791fbeaa0f0 [id=4e6695ccb7d342a6985b2a10a6bdfaac]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:47:55.665266 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 26d6f1105b324afba36081c760930ef0 because the tablet was not found. No further retry: Not found: Tablet not found: 26d6f1105b324afba36081c760930ef0
I20251024 19:47:55.665596 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e31eef151884385854096ff71484a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.666182 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 26d6f1105b324afba36081c760930ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:55.666896 1768 tablet_bootstrap.cc:492] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:55.667075 1768 ts_tablet_manager.cc:1403] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.006s
I20251024 19:47:55.667841 1768 raft_consensus.cc:359] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.668047 1768 raft_consensus.cc:385] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.668134 1768 raft_consensus.cc:740] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.668388 1768 consensus_queue.cc:260] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.665120 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 26d6f1105b324afba36081c760930ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:55.666870 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 0df102c2be684823a8eb3b9c5d40d744 (DEFAULT_TABLE table=default.Table_ae01a61e72ea4e9592e345869e67e586 [id=41f77affc9b0465d8159bbbc2d16646c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.670315 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0df102c2be684823a8eb3b9c5d40d744. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.670842 1768 ts_tablet_manager.cc:1434] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251024 19:47:55.672791 1768 tablet_bootstrap.cc:492] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:55.674437 1768 tablet_bootstrap.cc:654] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.675583 1767 raft_consensus.cc:359] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:47:55.679018 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 26d6f1105b324afba36081c760930ef0 because the tablet was not found. No further retry: Not found: Tablet not found: 26d6f1105b324afba36081c760930ef0
I20251024 19:47:55.679540 1695 heartbeater.cc:499] Master 127.0.252.252:36627 was elected leader, sending a full tablet report...
I20251024 19:47:55.680354 1767 raft_consensus.cc:385] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.680583 1767 raft_consensus.cc:740] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.681456 1767 consensus_queue.cc:260] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:47:55.683964 1179 catalog_manager.cc:4952] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 26d6f1105b324afba36081c760930ef0 because the tablet was not found. No further retry: Not found: Tablet not found: 26d6f1105b324afba36081c760930ef0
I20251024 19:47:55.686321 1627 tablet_service.cc:1505] Processing CreateTablet for tablet 9af5d5bd80a14f2ebdc7a595b17416e6 (DEFAULT_TABLE table=default.Table_73de54ad8b6b436097fe701dfd52d703 [id=10bb93537e8344b2bf08a46e2f1109a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.686761 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af5d5bd80a14f2ebdc7a595b17416e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.687047 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 9af5d5bd80a14f2ebdc7a595b17416e6 (DEFAULT_TABLE table=default.Table_73de54ad8b6b436097fe701dfd52d703 [id=10bb93537e8344b2bf08a46e2f1109a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.687443 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af5d5bd80a14f2ebdc7a595b17416e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.687366 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 9af5d5bd80a14f2ebdc7a595b17416e6 (DEFAULT_TABLE table=default.Table_73de54ad8b6b436097fe701dfd52d703 [id=10bb93537e8344b2bf08a46e2f1109a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.687829 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af5d5bd80a14f2ebdc7a595b17416e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.692905 1767 ts_tablet_manager.cc:1434] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.161s user 0.013s sys 0.003s
I20251024 19:47:55.693758 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 7e31eef151884385854096ff71484a9c (DEFAULT_TABLE table=default.Table_a66b74f191194717bd7ab791fbeaa0f0 [id=4e6695ccb7d342a6985b2a10a6bdfaac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.694139 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e31eef151884385854096ff71484a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.694837 1767 tablet_bootstrap.cc:492] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:55.696585 1767 tablet_bootstrap.cc:654] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.697324 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 44d777147a60462e86b3abf18a40f2c9 (DEFAULT_TABLE table=default.Table_7ddfa79850884c62900d2e92bc798c64 [id=750d9803961d4003b052aa142e097a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.697806 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44d777147a60462e86b3abf18a40f2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.703428 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7f9a6a1dc46744a385cdb484524c5c59
I20251024 19:47:55.703801 1183 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 5386938a8faf40e0ba1c7838bfbc35bc (table default.Table_8b6c3241e78148feadc4a16fc4c00582 [id=aadca5a1b6d4494aa37f8a8cc76ad12a]). Expected version 2 got 0
I20251024 19:47:55.703909 1189 catalog_manager.cc:6719] ReplaceTablet: tablet 7f9a6a1dc46744a385cdb484524c5c59 deleted and replaced by tablet f1ad65fd080e452e8094975d420336c7
I20251024 19:47:55.704399 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 74c8c9949988403890a96ca535d367aa
W20251024 19:47:55.704486 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1ad65fd080e452e8094975d420336c7 has no consensus state
I20251024 19:47:55.705116 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 0df102c2be684823a8eb3b9c5d40d744
I20251024 19:47:55.705183 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 74c8c9949988403890a96ca535d367aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:55.705732 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 20f32e34ce51446d8cc7d4938017e60e
I20251024 19:47:55.706219 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 44d777147a60462e86b3abf18a40f2c9 (DEFAULT_TABLE table=default.Table_7ddfa79850884c62900d2e92bc798c64 [id=750d9803961d4003b052aa142e097a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.706625 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44d777147a60462e86b3abf18a40f2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.708331 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f9a6a1dc46744a385cdb484524c5c59 with delete_type TABLET_DATA_DELETED (replaced by tablet f1ad65fd080e452e8094975d420336c7) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:55.709776 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 74c8c9949988403890a96ca535d367aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:55.711658 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 0df102c2be684823a8eb3b9c5d40d744 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:55.711771 1191 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 5386938a8faf40e0ba1c7838bfbc35bc (table default.Table_8b6c3241e78148feadc4a16fc4c00582 [id=aadca5a1b6d4494aa37f8a8cc76ad12a]). Expected version 2 got 0
I20251024 19:47:55.713037 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 20f32e34ce51446d8cc7d4938017e60e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:47:55.713954 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 74c8c9949988403890a96ca535d367aa because the tablet was not found. No further retry: Not found: Tablet not found: 74c8c9949988403890a96ca535d367aa
I20251024 19:47:55.714468 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet 20f32e34ce51446d8cc7d4938017e60e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:47:55.714567 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7f9a6a1dc46744a385cdb484524c5c59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f9a6a1dc46744a385cdb484524c5c59 already in progress: creating tablet
I20251024 19:47:55.714735 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet 2218f5fe8f77443992a9b83c694e2d20 with delete_type TABLET_DATA_DELETED (replaced by tablet 9af5d5bd80a14f2ebdc7a595b17416e6) from {username='slave'} at 127.0.0.1:54246
W20251024 19:47:55.714830 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 74c8c9949988403890a96ca535d367aa because the tablet was not found. No further retry: Not found: Tablet not found: 74c8c9949988403890a96ca535d367aa
I20251024 19:47:55.714964 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet f34c1080c5554c66b91c99ea294632b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 20f32e34ce51446d8cc7d4938017e60e) from {username='slave'} at 127.0.0.1:54246
W20251024 19:47:55.715019 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 0df102c2be684823a8eb3b9c5d40d744 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0df102c2be684823a8eb3b9c5d40d744 already in progress: creating tablet
W20251024 19:47:55.715224 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 20f32e34ce51446d8cc7d4938017e60e because the tablet was not found. No further retry: Not found: Tablet not found: 20f32e34ce51446d8cc7d4938017e60e
I20251024 19:47:55.715279 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet ad673e92efc94d33ad07a02b6464e724 with delete_type TABLET_DATA_DELETED (replaced by tablet 44d777147a60462e86b3abf18a40f2c9) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:55.715553 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 0df102c2be684823a8eb3b9c5d40d744 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:55.716166 1776 tablet_replica.cc:333] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:55.716343 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 74c8c9949988403890a96ca535d367aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:55.716567 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 20f32e34ce51446d8cc7d4938017e60e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:55.716632 1776 raft_consensus.cc:2243] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:55.717025 1776 raft_consensus.cc:2272] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:47:55.717031 1557 heartbeater.cc:646] Failed to heartbeat to 127.0.252.254:36505 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.252.254:36505 (error 108)
I20251024 19:47:55.717029 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 2218f5fe8f77443992a9b83c694e2d20 with delete_type TABLET_DATA_DELETED (replaced by tablet 9af5d5bd80a14f2ebdc7a595b17416e6) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:55.717546 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet 0df102c2be684823a8eb3b9c5d40d744 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:55.717764 1774 tablet_replica.cc:333] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:55.718173 1774 raft_consensus.cc:2243] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:47:55.718272 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 20f32e34ce51446d8cc7d4938017e60e because the tablet was not found. No further retry: Not found: Tablet not found: 20f32e34ce51446d8cc7d4938017e60e
W20251024 19:47:55.718328 1584 connection.cc:537] client connection to 127.0.252.254:36505 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251024 19:47:55.718536 1774 raft_consensus.cc:2272] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:47:55.718568 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2218f5fe8f77443992a9b83c694e2d20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2218f5fe8f77443992a9b83c694e2d20 already in progress: creating tablet
W20251024 19:47:55.718791 1696 heartbeater.cc:646] Failed to heartbeat to 127.0.252.254:36505 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251024 19:47:55.720062 1776 ts_tablet_manager.cc:1916] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:55.723254 1774 ts_tablet_manager.cc:1916] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:55.724846 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f9a6a1dc46744a385cdb484524c5c59 with delete_type TABLET_DATA_DELETED (replaced by tablet f1ad65fd080e452e8094975d420336c7) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:55.704879 1627 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f9a6a1dc46744a385cdb484524c5c59 with delete_type TABLET_DATA_DELETED (replaced by tablet f1ad65fd080e452e8094975d420336c7) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:55.726522 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 20f32e34ce51446d8cc7d4938017e60e (DEFAULT_TABLE table=default.Table_835ab1cc0a82456f8282bce320dca4f8 [id=91d53300d972437d9efba6385dbef72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.727030 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20f32e34ce51446d8cc7d4938017e60e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:55.732261 1418 heartbeater.cc:646] Failed to heartbeat to 127.0.252.254:36505 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.252.254:36505 (error 108)
W20251024 19:47:55.733187 1177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.252.254:36505 (error 108)
I20251024 19:47:55.734433 1627 tablet_service.cc:1505] Processing CreateTablet for tablet 44d777147a60462e86b3abf18a40f2c9 (DEFAULT_TABLE table=default.Table_7ddfa79850884c62900d2e92bc798c64 [id=750d9803961d4003b052aa142e097a45]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:47:55.734591 1777 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 19:47:55.732783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.252.254:36505 (local address 127.0.252.193:45241)
1024 19:47:55.732951 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1024 19:47:55.732967 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 19:47:55.733066 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 19:47:55.733703 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.252.254:36505: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":45}
I20251024 19:47:55.735008 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44d777147a60462e86b3abf18a40f2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.736333 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f9a6a1dc46744a385cdb484524c5c59 with delete_type TABLET_DATA_DELETED (replaced by tablet f1ad65fd080e452e8094975d420336c7) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:55.738008 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 20f32e34ce51446d8cc7d4938017e60e (DEFAULT_TABLE table=default.Table_835ab1cc0a82456f8282bce320dca4f8 [id=91d53300d972437d9efba6385dbef72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.738512 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20f32e34ce51446d8cc7d4938017e60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.738534 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 20f32e34ce51446d8cc7d4938017e60e (DEFAULT_TABLE table=default.Table_835ab1cc0a82456f8282bce320dca4f8 [id=91d53300d972437d9efba6385dbef72c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:47:55.738590 1420 tablet.cc:2378] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 19:47:55.739988 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 26d6f1105b324afba36081c760930ef0 (DEFAULT_TABLE table=default.Table_2759324fce09406ab532d22228541f6e [id=30be39f295db49d98ee0c102dd6b3b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.740130 1487 tablet_service.cc:1505] Processing CreateTablet for tablet f396c2982f8d487cb44c9422b09da14a (DEFAULT_TABLE table=default.Table_c4ffe38e352744a3891e5b1435036e9f [id=969d428d45db42d18a5a9b82ec945bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.740274 1768 tablet_bootstrap.cc:492] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:55.740415 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26d6f1105b324afba36081c760930ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.740420 1768 ts_tablet_manager.cc:1403] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.068s user 0.007s sys 0.007s
I20251024 19:47:55.740625 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f396c2982f8d487cb44c9422b09da14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.741247 1768 raft_consensus.cc:359] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.741479 1768 raft_consensus.cc:385] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.741565 1768 raft_consensus.cc:740] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.741818 1768 consensus_queue.cc:260] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.742291 1768 ts_tablet_manager.cc:1434] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:47:55.745249 1774 ts_tablet_manager.cc:1929] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:55.745389 1774 log.cc:1199] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/2218f5fe8f77443992a9b83c694e2d20
I20251024 19:47:55.747822 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 26d6f1105b324afba36081c760930ef0 (DEFAULT_TABLE table=default.Table_2759324fce09406ab532d22228541f6e [id=30be39f295db49d98ee0c102dd6b3b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.742972 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf7d13f588543b1973364fb7e801ffa (DEFAULT_TABLE table=default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.748322 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf7d13f588543b1973364fb7e801ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.748549 1776 ts_tablet_manager.cc:1929] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:55.748714 1776 log.cc:1199] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/f34c1080c5554c66b91c99ea294632b4
I20251024 19:47:55.749217 1776 ts_tablet_manager.cc:1950] T f34c1080c5554c66b91c99ea294632b4 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:55.752529 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26d6f1105b324afba36081c760930ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.743574 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 26d6f1105b324afba36081c760930ef0 (DEFAULT_TABLE table=default.Table_2759324fce09406ab532d22228541f6e [id=30be39f295db49d98ee0c102dd6b3b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.744086 1625 tablet_service.cc:1505] Processing CreateTablet for tablet f396c2982f8d487cb44c9422b09da14a (DEFAULT_TABLE table=default.Table_c4ffe38e352744a3891e5b1435036e9f [id=969d428d45db42d18a5a9b82ec945bfb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:47:55.755796 1179 catalog_manager.cc:4952] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 74c8c9949988403890a96ca535d367aa because the tablet was not found. No further retry: Not found: Tablet not found: 74c8c9949988403890a96ca535d367aa
W20251024 19:47:55.756079 1179 catalog_manager.cc:4952] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 20f32e34ce51446d8cc7d4938017e60e because the tablet was not found. No further retry: Not found: Tablet not found: 20f32e34ce51446d8cc7d4938017e60e
I20251024 19:47:55.756690 1768 tablet_bootstrap.cc:492] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:55.756842 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet ad673e92efc94d33ad07a02b6464e724 with delete_type TABLET_DATA_DELETED (replaced by tablet 44d777147a60462e86b3abf18a40f2c9) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:55.757355 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf7d13f588543b1973364fb7e801ffa (DEFAULT_TABLE table=default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.757732 1774 ts_tablet_manager.cc:1950] T 2218f5fe8f77443992a9b83c694e2d20 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:55.757807 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf7d13f588543b1973364fb7e801ffa. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:55.758359 1177 consensus_peers.cc:597] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c -> Peer ef8906492b1a4368936fac95c11e1f73 (127.0.252.254:36505): Couldn't send request to peer ef8906492b1a4368936fac95c11e1f73. Status: Network error: Client connection negotiation failed: client connection to 127.0.252.254:36505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251024 19:47:55.758433 1768 tablet_bootstrap.cc:654] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.760124 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26d6f1105b324afba36081c760930ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.760900 1767 tablet_bootstrap.cc:492] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:55.761118 1767 ts_tablet_manager.cc:1403] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.007s
I20251024 19:47:55.761898 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf7d13f588543b1973364fb7e801ffa (DEFAULT_TABLE table=default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.762326 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf7d13f588543b1973364fb7e801ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.762428 1767 raft_consensus.cc:359] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.762697 1767 raft_consensus.cc:385] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.762851 1767 raft_consensus.cc:740] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.763201 1767 consensus_queue.cc:260] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.766211 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c5b5dcab83194aa28ae6465fbaa7f023"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.766811 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5b5dcab83194aa28ae6465fbaa7f023 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.767537 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 2218f5fe8f77443992a9b83c694e2d20 with delete_type TABLET_DATA_DELETED (replaced by tablet 9af5d5bd80a14f2ebdc7a595b17416e6) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:55.768299 1766 ts_tablet_manager.cc:1434] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.222s user 0.009s sys 0.002s
I20251024 19:47:55.768857 1766 tablet_bootstrap.cc:492] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:47:55.769392 1784 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 19:47:55.717528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.252.254:36505 (local address 127.0.252.194:33669)
1024 19:47:55.731761 (+ 14233us) negotiation.cc:107] Waiting for socket to connect
1024 19:47:55.731785 (+ 24us) client_negotiation.cc:174] Beginning negotiation
1024 19:47:55.731921 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 19:47:55.768294 (+ 36373us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.252.254:36505: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":14074,"thread_start_us":13990,"threads_started":1}
I20251024 19:47:55.770066 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet 5386938a8faf40e0ba1c7838bfbc35bc from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:55.772333 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet 44d777147a60462e86b3abf18a40f2c9 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:55.773408 1766 tablet_bootstrap.cc:654] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.773447 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f396c2982f8d487cb44c9422b09da14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.773821 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20f32e34ce51446d8cc7d4938017e60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.775986 1767 ts_tablet_manager.cc:1434] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251024 19:47:55.777484 1767 tablet_bootstrap.cc:492] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:47:55.778333 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7f9a6a1dc46744a385cdb484524c5c59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f9a6a1dc46744a385cdb484524c5c59 already in progress: creating tablet
I20251024 19:47:55.779176 1767 tablet_bootstrap.cc:654] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.780915 1628 tablet_service.cc:1505] Processing CreateTablet for tablet b96722ee06cf4e26a55a913c060a1209 (DEFAULT_TABLE table=default.Table_e7a8fc2ba8e0481cb7479d33f3ecbdf2 [id=350b37bf42194fcfbb660bffe83e47c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.781376 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b96722ee06cf4e26a55a913c060a1209. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:55.782061 1785 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 19:47:55.720465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.252.254:36505 (local address 127.0.252.195:51945)
1024 19:47:55.732365 (+ 11900us) negotiation.cc:107] Waiting for socket to connect
1024 19:47:55.732384 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1024 19:47:55.732529 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 19:47:55.765051 (+ 32522us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.252.254:36505: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":11777,"thread_start_us":12153,"threads_started":1}
I20251024 19:47:55.784019 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 5386938a8faf40e0ba1c7838bfbc35bc
I20251024 19:47:55.784569 1183 catalog_manager.cc:6719] ReplaceTablet: tablet 5386938a8faf40e0ba1c7838bfbc35bc deleted and replaced by tablet 07d66fe64bff4a00a15554f212456698
I20251024 19:47:55.787747 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet 5386938a8faf40e0ba1c7838bfbc35bc with delete_type TABLET_DATA_DELETED (replaced by tablet 07d66fe64bff4a00a15554f212456698) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:55.788825 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 5386938a8faf40e0ba1c7838bfbc35bc with delete_type TABLET_DATA_DELETED (replaced by tablet 07d66fe64bff4a00a15554f212456698) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:55.788954 1351 tablet_service.cc:1505] Processing CreateTablet for tablet b96722ee06cf4e26a55a913c060a1209 (DEFAULT_TABLE table=default.Table_e7a8fc2ba8e0481cb7479d33f3ecbdf2 [id=350b37bf42194fcfbb660bffe83e47c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.789330 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b96722ee06cf4e26a55a913c060a1209. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.789789 1490 tablet_service.cc:1505] Processing CreateTablet for tablet b96722ee06cf4e26a55a913c060a1209 (DEFAULT_TABLE table=default.Table_e7a8fc2ba8e0481cb7479d33f3ecbdf2 [id=350b37bf42194fcfbb660bffe83e47c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.790220 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b96722ee06cf4e26a55a913c060a1209. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.790241 1768 tablet_bootstrap.cc:492] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:55.790428 1768 ts_tablet_manager.cc:1403] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20251024 19:47:55.791301 1768 raft_consensus.cc:359] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.791531 1768 raft_consensus.cc:385] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.791626 1768 raft_consensus.cc:740] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.791909 1768 consensus_queue.cc:260] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.794140 1768 ts_tablet_manager.cc:1434] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251024 19:47:55.794492 1768 tablet_bootstrap.cc:492] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:55.796003 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 74c8c9949988403890a96ca535d367aa (DEFAULT_TABLE table=default.Table_36406152efad4ca4b1a106ed81d39fbe [id=6d9d362a97bd4c6ba8ccc960082ed831]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.796386 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c8c9949988403890a96ca535d367aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.797919 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f9a6a1dc46744a385cdb484524c5c59 with delete_type TABLET_DATA_DELETED (replaced by tablet f1ad65fd080e452e8094975d420336c7) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:55.798920 1775 tablet_replica.cc:333] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:55.799413 1775 raft_consensus.cc:2243] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:55.799841 1775 raft_consensus.cc:2272] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:55.800637 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 44d777147a60462e86b3abf18a40f2c9
I20251024 19:47:55.801690 1188 catalog_manager.cc:6719] ReplaceTablet: tablet 44d777147a60462e86b3abf18a40f2c9 deleted and replaced by tablet c70e51a4bd7c423aae7d6e2ddf0c60b4
I20251024 19:47:55.801951 1627 tablet_service.cc:1552] Processing DeleteTablet for tablet 44d777147a60462e86b3abf18a40f2c9 with delete_type TABLET_DATA_DELETED (replaced by tablet c70e51a4bd7c423aae7d6e2ddf0c60b4) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:55.801944 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 74c8c9949988403890a96ca535d367aa (DEFAULT_TABLE table=default.Table_36406152efad4ca4b1a106ed81d39fbe [id=6d9d362a97bd4c6ba8ccc960082ed831]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.802116 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 74c8c9949988403890a96ca535d367aa (DEFAULT_TABLE table=default.Table_36406152efad4ca4b1a106ed81d39fbe [id=6d9d362a97bd4c6ba8ccc960082ed831]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.802322 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c8c9949988403890a96ca535d367aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.802455 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c8c9949988403890a96ca535d367aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.806512 1775 ts_tablet_manager.cc:1916] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:55.807000 1768 tablet_bootstrap.cc:654] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.809793 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 5386938a8faf40e0ba1c7838bfbc35bc with delete_type TABLET_DATA_DELETED (replaced by tablet 07d66fe64bff4a00a15554f212456698) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:55.810187 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 44d777147a60462e86b3abf18a40f2c9 with delete_type TABLET_DATA_DELETED (replaced by tablet c70e51a4bd7c423aae7d6e2ddf0c60b4) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:55.814711 1774 tablet_replica.cc:333] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:55.814812 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 2218f5fe8f77443992a9b83c694e2d20 (table default.Table_73de54ad8b6b436097fe701dfd52d703 [id=10bb93537e8344b2bf08a46e2f1109a9]) successfully deleted
I20251024 19:47:55.814996 1774 raft_consensus.cc:2243] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:55.815191 1774 raft_consensus.cc:2272] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:55.815549 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e35b6fdad5474d488ff1519ff3a63da1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.816191 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e35b6fdad5474d488ff1519ff3a63da1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.816761 1776 tablet_replica.cc:333] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:55.816962 1776 raft_consensus.cc:2243] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:55.817128 1776 raft_consensus.cc:2272] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:55.817194 1774 ts_tablet_manager.cc:1916] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:55.817646 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 44d777147a60462e86b3abf18a40f2c9 with delete_type TABLET_DATA_DELETED (replaced by tablet c70e51a4bd7c423aae7d6e2ddf0c60b4) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:55.818552 1775 ts_tablet_manager.cc:1929] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:55.818775 1775 log.cc:1199] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/7f9a6a1dc46744a385cdb484524c5c59
I20251024 19:47:55.818933 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet f34c1080c5554c66b91c99ea294632b4 (table default.Table_835ab1cc0a82456f8282bce320dca4f8 [id=91d53300d972437d9efba6385dbef72c]) successfully deleted
W20251024 19:47:55.819197 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet ad673e92efc94d33ad07a02b6464e724 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad673e92efc94d33ad07a02b6464e724 already in progress: creating tablet
I20251024 19:47:55.819445 1775 ts_tablet_manager.cc:1950] T 7f9a6a1dc46744a385cdb484524c5c59 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
W20251024 19:47:55.819445 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 0df102c2be684823a8eb3b9c5d40d744 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0df102c2be684823a8eb3b9c5d40d744 already in progress: creating tablet
W20251024 19:47:55.819697 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7f9a6a1dc46744a385cdb484524c5c59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f9a6a1dc46744a385cdb484524c5c59 already in progress: creating tablet
I20251024 19:47:55.820329 1776 ts_tablet_manager.cc:1916] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:55.822777 1352 tablet_service.cc:1505] Processing CreateTablet for tablet f396c2982f8d487cb44c9422b09da14a (DEFAULT_TABLE table=default.Table_c4ffe38e352744a3891e5b1435036e9f [id=969d428d45db42d18a5a9b82ec945bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.823215 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f396c2982f8d487cb44c9422b09da14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.823249 1774 ts_tablet_manager.cc:1929] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:55.823395 1774 log.cc:1199] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/7f9a6a1dc46744a385cdb484524c5c59
I20251024 19:47:55.823776 1774 ts_tablet_manager.cc:1950] T 7f9a6a1dc46744a385cdb484524c5c59 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:55.825794 1768 tablet_bootstrap.cc:492] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:55.825945 1768 ts_tablet_manager.cc:1403] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.003s
I20251024 19:47:55.826730 1768 raft_consensus.cc:359] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:55.826951 1768 raft_consensus.cc:385] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.827054 1768 raft_consensus.cc:740] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.827302 1768 consensus_queue.cc:260] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:55.830685 1768 ts_tablet_manager.cc:1434] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:47:55.830771 1774 tablet_replica.cc:333] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:55.830732 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_01b0d0ffefb744f3a1abe0eeab079dc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.830920 1774 raft_consensus.cc:2243] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:55.831090 1774 raft_consensus.cc:2272] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:47:55.831266 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01b0d0ffefb744f3a1abe0eeab079dc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.831879 1768 tablet_bootstrap.cc:492] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:55.832144 1774 ts_tablet_manager.cc:1916] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:55.835677 1767 tablet_bootstrap.cc:492] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:55.835863 1767 ts_tablet_manager.cc:1403] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.004s
I20251024 19:47:55.836208 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet f34c1080c5554c66b91c99ea294632b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 20f32e34ce51446d8cc7d4938017e60e) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:55.836576 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_cfcdb21fd4cb4126b99c081c711d490e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.836681 1767 raft_consensus.cc:359] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:55.836884 1767 raft_consensus.cc:385] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.836985 1767 raft_consensus.cc:740] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
W20251024 19:47:55.837049 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfcdb21fd4cb4126b99c081c711d490e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.837235 1767 consensus_queue.cc:260] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:55.837779 1767 ts_tablet_manager.cc:1434] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251024 19:47:55.839509 1767 tablet_bootstrap.cc:492] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:55.838022 1768 tablet_bootstrap.cc:654] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.841305 1767 tablet_bootstrap.cc:654] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.841711 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_cf69470650f643c195eb5fbecbd2d2c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.841908 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a2fc8ac42bbc44af8bf194f2d3511bd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.841996 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a35e66c1df02471480ce2e87424f6e91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.842199 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf69470650f643c195eb5fbecbd2d2c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:55.842437 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2fc8ac42bbc44af8bf194f2d3511bd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.843235 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_88ac0b5d20a946f9948436719299d223"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:55.843627 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8d9e77c4fcbf4a6e9f90d1c111a8cb50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.844099 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d9e77c4fcbf4a6e9f90d1c111a8cb50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.844805 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet 44d777147a60462e86b3abf18a40f2c9 from {username='slave'} at 127.0.0.1:51416
W20251024 19:47:55.843631 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88ac0b5d20a946f9948436719299d223 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.845384 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8ca12a13ba4742478879c608bd9ddf3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.845928 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ca12a13ba4742478879c608bd9ddf3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:55.842442 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a35e66c1df02471480ce2e87424f6e91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:55.847718 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet 5386938a8faf40e0ba1c7838bfbc35bc from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:55.850497 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e7a8fc2ba8e0481cb7479d33f3ecbdf2" } new_table_name: "default.Table_4dda32c7ff144630bebd572cf5f9f822" modify_external_catalogs: true
W20251024 19:47:55.852260 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 5386938a8faf40e0ba1c7838bfbc35bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5386938a8faf40e0ba1c7838bfbc35bc already in progress: creating tablet
I20251024 19:47:55.852684 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 7f9a6a1dc46744a385cdb484524c5c59 (table default.Table_7fb527771f604f21beccb4abca21a38f [id=2fd5c541b06a46469e65ed0942d2c394]) successfully deleted
I20251024 19:47:55.853047 1776 ts_tablet_manager.cc:1929] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:55.853204 1776 log.cc:1199] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/2218f5fe8f77443992a9b83c694e2d20
I20251024 19:47:55.853613 1776 ts_tablet_manager.cc:1950] T 2218f5fe8f77443992a9b83c694e2d20 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
W20251024 19:47:55.853889 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 44d777147a60462e86b3abf18a40f2c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44d777147a60462e86b3abf18a40f2c9 already in progress: creating tablet
I20251024 19:47:55.854672 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 2218f5fe8f77443992a9b83c694e2d20 (table default.Table_73de54ad8b6b436097fe701dfd52d703 [id=10bb93537e8344b2bf08a46e2f1109a9]) successfully deleted
W20251024 19:47:55.854936 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 44d777147a60462e86b3abf18a40f2c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44d777147a60462e86b3abf18a40f2c9 already in progress: creating tablet
W20251024 19:47:55.855175 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 5386938a8faf40e0ba1c7838bfbc35bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5386938a8faf40e0ba1c7838bfbc35bc already in progress: creating tablet
I20251024 19:47:55.858394 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7fb527771f604f21beccb4abca21a38f" } new_table_name: "default.Table_c4539f71581f488cad3bc3b4b2f3a0eb" modify_external_catalogs: true
I20251024 19:47:55.859622 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_73de54ad8b6b436097fe701dfd52d703" } new_table_name: "default.Table_08dbb5c3a0924b1491e302cef50518c4" modify_external_catalogs: true
W20251024 19:47:55.861400 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 0df102c2be684823a8eb3b9c5d40d744 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0df102c2be684823a8eb3b9c5d40d744 already in progress: creating tablet
I20251024 19:47:55.862031 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 7f9a6a1dc46744a385cdb484524c5c59 (table default.Table_7fb527771f604f21beccb4abca21a38f [id=2fd5c541b06a46469e65ed0942d2c394]) successfully deleted
I20251024 19:47:55.862424 1774 ts_tablet_manager.cc:1929] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:55.862586 1774 log.cc:1199] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/ad673e92efc94d33ad07a02b6464e724
I20251024 19:47:55.863032 1774 ts_tablet_manager.cc:1950] T ad673e92efc94d33ad07a02b6464e724 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:55.863515 1768 tablet_bootstrap.cc:492] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:55.863714 1768 ts_tablet_manager.cc:1403] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20251024 19:47:55.864159 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet ad673e92efc94d33ad07a02b6464e724 (table default.Table_7ddfa79850884c62900d2e92bc798c64 [id=750d9803961d4003b052aa142e097a45]) successfully deleted
I20251024 19:47:55.864332 1766 tablet_bootstrap.cc:492] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:55.864486 1774 tablet_replica.cc:333] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:55.864706 1774 raft_consensus.cc:2243] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:55.864636 1768 raft_consensus.cc:359] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:55.864861 1768 raft_consensus.cc:385] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.864886 1774 raft_consensus.cc:2272] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:55.864953 1768 raft_consensus.cc:740] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.864976 1766 ts_tablet_manager.cc:1403] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.013s
I20251024 19:47:55.865229 1768 consensus_queue.cc:260] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:55.866077 1774 ts_tablet_manager.cc:1916] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:55.866273 1766 raft_consensus.cc:359] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.866528 1766 raft_consensus.cc:385] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.866683 1766 raft_consensus.cc:740] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.867007 1767 tablet_bootstrap.cc:492] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:55.867163 1767 ts_tablet_manager.cc:1403] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.005s
I20251024 19:47:55.867107 1766 consensus_queue.cc:260] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.867996 1767 raft_consensus.cc:359] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.868216 1767 raft_consensus.cc:385] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.868314 1767 raft_consensus.cc:740] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.868577 1767 consensus_queue.cc:260] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.868813 1768 ts_tablet_manager.cc:1434] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251024 19:47:55.871068 1766 ts_tablet_manager.cc:1434] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251024 19:47:55.871620 1767 ts_tablet_manager.cc:1434] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251024 19:47:55.871685 1768 tablet_bootstrap.cc:492] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:55.872314 1774 ts_tablet_manager.cc:1929] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:55.872547 1774 log.cc:1199] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/5386938a8faf40e0ba1c7838bfbc35bc
I20251024 19:47:55.872823 1767 tablet_bootstrap.cc:492] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:55.873030 1774 ts_tablet_manager.cc:1950] T 5386938a8faf40e0ba1c7838bfbc35bc P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:55.873714 1768 tablet_bootstrap.cc:654] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.874137 1766 tablet_bootstrap.cc:492] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:55.874532 1774 tablet_replica.cc:333] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:55.874636 1767 tablet_bootstrap.cc:654] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.874717 1774 raft_consensus.cc:2243] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:55.874912 1774 raft_consensus.cc:2272] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:55.875176 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 5386938a8faf40e0ba1c7838bfbc35bc (table default.Table_8b6c3241e78148feadc4a16fc4c00582 [id=aadca5a1b6d4494aa37f8a8cc76ad12a]) successfully deleted
I20251024 19:47:55.875630 1774 ts_tablet_manager.cc:1916] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:47:55.875948 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 44d777147a60462e86b3abf18a40f2c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44d777147a60462e86b3abf18a40f2c9 already in progress: creating tablet
I20251024 19:47:55.877478 1766 tablet_bootstrap.cc:654] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.881155 1767 tablet_bootstrap.cc:492] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:55.881376 1767 ts_tablet_manager.cc:1403] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20251024 19:47:55.881969 1774 ts_tablet_manager.cc:1929] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:55.882102 1774 log.cc:1199] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/f34c1080c5554c66b91c99ea294632b4
I20251024 19:47:55.882352 1767 raft_consensus.cc:359] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.882520 1774 ts_tablet_manager.cc:1950] T f34c1080c5554c66b91c99ea294632b4 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:55.882627 1767 raft_consensus.cc:385] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.882710 1767 raft_consensus.cc:740] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.883014 1767 consensus_queue.cc:260] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.883387 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet f34c1080c5554c66b91c99ea294632b4 (table default.Table_835ab1cc0a82456f8282bce320dca4f8 [id=91d53300d972437d9efba6385dbef72c]) successfully deleted
I20251024 19:47:55.884459 1768 tablet_bootstrap.cc:492] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:55.884608 1768 ts_tablet_manager.cc:1403] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20251024 19:47:55.885329 1768 raft_consensus.cc:359] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.885535 1768 raft_consensus.cc:385] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.885654 1768 raft_consensus.cc:740] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.885862 1768 consensus_queue.cc:260] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.886361 1768 ts_tablet_manager.cc:1434] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:47:55.886655 1768 tablet_bootstrap.cc:492] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:55.886868 1766 tablet_bootstrap.cc:492] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:55.887082 1766 ts_tablet_manager.cc:1403] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20251024 19:47:55.887528 1767 ts_tablet_manager.cc:1434] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20251024 19:47:55.888199 1768 tablet_bootstrap.cc:654] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.888134 1766 raft_consensus.cc:359] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.888458 1766 raft_consensus.cc:385] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.888576 1766 raft_consensus.cc:740] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.888921 1766 consensus_queue.cc:260] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.890168 1767 tablet_bootstrap.cc:492] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:55.891405 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c5b5dcab83194aa28ae6465fbaa7f023" } modify_external_catalogs: true
I20251024 19:47:55.891625 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c5b5dcab83194aa28ae6465fbaa7f023" } modify_external_catalogs: true
I20251024 19:47:55.891773 1767 tablet_bootstrap.cc:654] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.892617 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a66b74f191194717bd7ab791fbeaa0f0" } new_table_name: "default.Table_dabc8d67782e4372b14bcf1862b25d3a" modify_external_catalogs: true
I20251024 19:47:55.892717 1766 ts_tablet_manager.cc:1434] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251024 19:47:55.894735 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7ddfa79850884c62900d2e92bc798c64" } modify_external_catalogs: true
I20251024 19:47:55.894933 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7ddfa79850884c62900d2e92bc798c64" } modify_external_catalogs: true
I20251024 19:47:55.897014 1766 tablet_bootstrap.cc:492] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:55.897946 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 4ed6c66a4ce141fbabbee58877689439 (DEFAULT_TABLE table=default.Table_c5b5dcab83194aa28ae6465fbaa7f023 [id=c112bf97e1fa4c21ac91f769bdfc6f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.898412 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed6c66a4ce141fbabbee58877689439. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.899569 1766 tablet_bootstrap.cc:654] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:47:55.901913 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b96722ee06cf4e26a55a913c060a1209 consensus state has no leader
I20251024 19:47:55.902225 1629 tablet_service.cc:1505] Processing CreateTablet for tablet fab45ee691fe495196cf671e69368d4d (DEFAULT_TABLE table=default.Table_46d6218260e8421b9093ea1630d5c8de [id=79ccc02d30c743d0a448971d552b35dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.902688 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab45ee691fe495196cf671e69368d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.903509 1766 tablet_bootstrap.cc:492] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:55.903631 1766 ts_tablet_manager.cc:1403] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20251024 19:47:55.904467 1766 raft_consensus.cc:359] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.904644 1766 raft_consensus.cc:385] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.904735 1766 raft_consensus.cc:740] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.904968 1766 consensus_queue.cc:260] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.905826 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 07d66fe64bff4a00a15554f212456698 (DEFAULT_TABLE table=default.Table_8b6c3241e78148feadc4a16fc4c00582 [id=aadca5a1b6d4494aa37f8a8cc76ad12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.906214 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07d66fe64bff4a00a15554f212456698. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.907205 1745 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:47:55.907461 1745 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: This master's current role is: LEADER
I20251024 19:47:55.908196 1766 ts_tablet_manager.cc:1434] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:47:55.909832 1625 tablet_service.cc:1505] Processing CreateTablet for tablet f1ad65fd080e452e8094975d420336c7 (DEFAULT_TABLE table=default.Table_7fb527771f604f21beccb4abca21a38f [id=2fd5c541b06a46469e65ed0942d2c394]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.910164 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1ad65fd080e452e8094975d420336c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.911484 1351 tablet_service.cc:1505] Processing CreateTablet for tablet fab45ee691fe495196cf671e69368d4d (DEFAULT_TABLE table=default.Table_46d6218260e8421b9093ea1630d5c8de [id=79ccc02d30c743d0a448971d552b35dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.911921 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab45ee691fe495196cf671e69368d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.912509 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 4ed6c66a4ce141fbabbee58877689439 (DEFAULT_TABLE table=default.Table_c5b5dcab83194aa28ae6465fbaa7f023 [id=c112bf97e1fa4c21ac91f769bdfc6f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.912885 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed6c66a4ce141fbabbee58877689439. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.913933 1350 tablet_service.cc:1505] Processing CreateTablet for tablet f1ad65fd080e452e8094975d420336c7 (DEFAULT_TABLE table=default.Table_7fb527771f604f21beccb4abca21a38f [id=2fd5c541b06a46469e65ed0942d2c394]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:47:55.913997 1698 tablet.cc:2378] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 19:47:55.914386 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1ad65fd080e452e8094975d420336c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.915263 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 4ed6c66a4ce141fbabbee58877689439 (DEFAULT_TABLE table=default.Table_c5b5dcab83194aa28ae6465fbaa7f023 [id=c112bf97e1fa4c21ac91f769bdfc6f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.915612 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed6c66a4ce141fbabbee58877689439. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.916407 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 7e31eef151884385854096ff71484a9c (table default.Table_a66b74f191194717bd7ab791fbeaa0f0 [id=4e6695ccb7d342a6985b2a10a6bdfaac]). Expected version 1 got 0
I20251024 19:47:55.916656 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 9af5d5bd80a14f2ebdc7a595b17416e6 (table default.Table_73de54ad8b6b436097fe701dfd52d703 [id=10bb93537e8344b2bf08a46e2f1109a9]). Expected version 2 got 0
I20251024 19:47:55.917901 1489 tablet_service.cc:1505] Processing CreateTablet for tablet fab45ee691fe495196cf671e69368d4d (DEFAULT_TABLE table=default.Table_46d6218260e8421b9093ea1630d5c8de [id=79ccc02d30c743d0a448971d552b35dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.918072 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8b6c3241e78148feadc4a16fc4c00582" } new_table_name: "default.Table_7c34d3a8affd4e69a31b276ccbe096ae" modify_external_catalogs: true
I20251024 19:47:55.918249 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab45ee691fe495196cf671e69368d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.918418 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 5386938a8faf40e0ba1c7838bfbc35bc with delete_type TABLET_DATA_DELETED (replaced by tablet 07d66fe64bff4a00a15554f212456698) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:55.918800 1487 tablet_service.cc:1505] Processing CreateTablet for tablet f1ad65fd080e452e8094975d420336c7 (DEFAULT_TABLE table=default.Table_7fb527771f604f21beccb4abca21a38f [id=2fd5c541b06a46469e65ed0942d2c394]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.919175 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1ad65fd080e452e8094975d420336c7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:55.920773 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1ad65fd080e452e8094975d420336c7 consensus state has no leader
I20251024 19:47:55.921849 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 07d66fe64bff4a00a15554f212456698 (DEFAULT_TABLE table=default.Table_8b6c3241e78148feadc4a16fc4c00582 [id=aadca5a1b6d4494aa37f8a8cc76ad12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.922210 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07d66fe64bff4a00a15554f212456698. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:55.922160 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e35b6fdad5474d488ff1519ff3a63da1" } modify_external_catalogs: true
I20251024 19:47:55.922326 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e35b6fdad5474d488ff1519ff3a63da1" } modify_external_catalogs: true
I20251024 19:47:55.923386 1775 tablet_replica.cc:333] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:55.923486 1782 raft_consensus.cc:493] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:55.923600 1775 raft_consensus.cc:2243] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:55.923648 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_46d6218260e8421b9093ea1630d5c8de" } modify_external_catalogs: true
I20251024 19:47:55.923765 1775 raft_consensus.cc:2272] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:55.923717 1782 raft_consensus.cc:515] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:55.923846 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_46d6218260e8421b9093ea1630d5c8de" } modify_external_catalogs: true
I20251024 19:47:55.925318 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cfcdb21fd4cb4126b99c081c711d490e" } modify_external_catalogs: true
I20251024 19:47:55.925524 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cfcdb21fd4cb4126b99c081c711d490e" } modify_external_catalogs: true
I20251024 19:47:55.925729 1782 leader_election.cc:290] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:47:55.927780 1766 tablet_bootstrap.cc:492] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:55.929522 1766 tablet_bootstrap.cc:654] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.931386 1775 ts_tablet_manager.cc:1916] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:55.937721 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 07d66fe64bff4a00a15554f212456698 (DEFAULT_TABLE table=default.Table_8b6c3241e78148feadc4a16fc4c00582 [id=aadca5a1b6d4494aa37f8a8cc76ad12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:55.938277 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07d66fe64bff4a00a15554f212456698. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:55.938371 1586 leader_election.cc:343] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 7f9a6a1dc46744a385cdb484524c5c59
W20251024 19:47:55.939819 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9af5d5bd80a14f2ebdc7a595b17416e6 consensus state has no leader
I20251024 19:47:55.941898 1775 ts_tablet_manager.cc:1929] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:55.942099 1775 log.cc:1199] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/5386938a8faf40e0ba1c7838bfbc35bc
I20251024 19:47:55.942505 1775 ts_tablet_manager.cc:1950] T 5386938a8faf40e0ba1c7838bfbc35bc P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:55.943542 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 5386938a8faf40e0ba1c7838bfbc35bc (table default.Table_8b6c3241e78148feadc4a16fc4c00582 [id=aadca5a1b6d4494aa37f8a8cc76ad12a]) successfully deleted
W20251024 19:47:55.944811 1177 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.253:41375, user_credentials={real_user=slave}} blocked reactor thread for 60770us
I20251024 19:47:55.964323 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet 9af5d5bd80a14f2ebdc7a595b17416e6 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:55.967269 1782 raft_consensus.cc:493] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:55.967525 1782 raft_consensus.cc:515] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.968274 1782 leader_election.cc:290] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:47:55.969453 1586 leader_election.cc:343] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 5386938a8faf40e0ba1c7838bfbc35bc
I20251024 19:47:55.969684 1766 tablet_bootstrap.cc:492] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:55.970135 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f9a6a1dc46744a385cdb484524c5c59 with delete_type TABLET_DATA_DELETED (replaced by tablet f1ad65fd080e452e8094975d420336c7) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:55.970544 1776 tablet_replica.cc:333] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:55.970731 1776 raft_consensus.cc:2243] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:55.970911 1776 raft_consensus.cc:2272] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:55.971740 1776 ts_tablet_manager.cc:1916] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:47:55.970549 1586 leader_election.cc:343] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 7f9a6a1dc46744a385cdb484524c5c59
I20251024 19:47:55.971918 1586 leader_election.cc:304] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
W20251024 19:47:55.972316 1586 raft_consensus.cc:2686] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251024 19:47:55.972775 1586 leader_election.cc:343] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 5386938a8faf40e0ba1c7838bfbc35bc
I20251024 19:47:55.972900 1586 leader_election.cc:304] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:47:55.973264 1766 ts_tablet_manager.cc:1403] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.000s
I20251024 19:47:55.974153 1766 raft_consensus.cc:359] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.974342 1766 raft_consensus.cc:385] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.974406 1766 raft_consensus.cc:740] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.974684 1766 consensus_queue.cc:260] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:55.977437 1766 ts_tablet_manager.cc:1434] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:47:55.978528 1766 tablet_bootstrap.cc:492] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:55.980149 1766 tablet_bootstrap.cc:654] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:55.989769 1782 raft_consensus.cc:2749] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:55.992424 1776 ts_tablet_manager.cc:1929] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:55.992671 1776 log.cc:1199] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/7f9a6a1dc46744a385cdb484524c5c59
I20251024 19:47:55.993242 1776 ts_tablet_manager.cc:1950] T 7f9a6a1dc46744a385cdb484524c5c59 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:55.994652 1767 tablet_bootstrap.cc:492] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:55.994902 1767 ts_tablet_manager.cc:1403] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.105s user 0.002s sys 0.006s
W20251024 19:47:55.995573 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e31eef151884385854096ff71484a9c consensus state has no leader
I20251024 19:47:55.995863 1767 raft_consensus.cc:359] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:55.996070 1767 raft_consensus.cc:385] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:55.996163 1767 raft_consensus.cc:740] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:55.997753 1192 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet c70e51a4bd7c423aae7d6e2ddf0c60b4
I20251024 19:47:55.998207 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6b2b5091b93348958149ec3640ed6971"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:55.998927 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b2b5091b93348958149ec3640ed6971 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:56.000495 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07d66fe64bff4a00a15554f212456698 consensus state has no leader
I20251024 19:47:56.000787 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet af2cc8c940b94b0ab10bfc98c8618e6f from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.001211 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet fab45ee691fe495196cf671e69368d4d
I20251024 19:47:56.001920 1184 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 5e8a8325e77945ea808deba455f394ed
I20251024 19:47:56.002277 1768 tablet_bootstrap.cc:492] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.002522 1768 ts_tablet_manager.cc:1403] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.116s user 0.052s sys 0.052s
I20251024 19:47:56.002561 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet fab45ee691fe495196cf671e69368d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.002985 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet fab45ee691fe495196cf671e69368d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:47:56.003338 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet fab45ee691fe495196cf671e69368d4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fab45ee691fe495196cf671e69368d4d already in progress: creating tablet
I20251024 19:47:56.003960 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet fab45ee691fe495196cf671e69368d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.004699 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 7f9a6a1dc46744a385cdb484524c5c59 (table default.Table_c4539f71581f488cad3bc3b4b2f3a0eb [id=2fd5c541b06a46469e65ed0942d2c394]) successfully deleted
I20251024 19:47:56.004791 1767 consensus_queue.cc:260] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:47:56.005223 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet fab45ee691fe495196cf671e69368d4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fab45ee691fe495196cf671e69368d4d already in progress: creating tablet
I20251024 19:47:56.006580 1190 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet ba07b1061d1d4ad38eaf765123082f8e
I20251024 19:47:56.006634 1767 ts_tablet_manager.cc:1434] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251024 19:47:56.006991 1767 tablet_bootstrap.cc:492] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:56.007396 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4ed6c66a4ce141fbabbee58877689439
I20251024 19:47:56.008767 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ed6c66a4ce141fbabbee58877689439 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.009438 1767 tablet_bootstrap.cc:654] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:47:56.009934 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4ed6c66a4ce141fbabbee58877689439 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ed6c66a4ce141fbabbee58877689439 already in progress: creating tablet
I20251024 19:47:56.003378 1768 raft_consensus.cc:359] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.011143 1768 raft_consensus.cc:385] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.011262 1768 raft_consensus.cc:740] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
W20251024 19:47:56.011605 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet fab45ee691fe495196cf671e69368d4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fab45ee691fe495196cf671e69368d4d already in progress: creating tablet
I20251024 19:47:56.011592 1768 consensus_queue.cc:260] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.011897 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet 5386938a8faf40e0ba1c7838bfbc35bc with delete_type TABLET_DATA_DELETED (replaced by tablet 07d66fe64bff4a00a15554f212456698) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.012266 1768 ts_tablet_manager.cc:1434] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251024 19:47:56.012753 1768 tablet_bootstrap.cc:492] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.012934 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet ad673e92efc94d33ad07a02b6464e724 with delete_type TABLET_DATA_DELETED (replaced by tablet 44d777147a60462e86b3abf18a40f2c9) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.013142 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f9a6a1dc46744a385cdb484524c5c59 with delete_type TABLET_DATA_DELETED (replaced by tablet f1ad65fd080e452e8094975d420336c7) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.013248 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ed6c66a4ce141fbabbee58877689439 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.013491 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ed6c66a4ce141fbabbee58877689439 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.014995 1768 tablet_bootstrap.cc:654] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.015105 1776 tablet_replica.cc:333] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:56.015300 1776 raft_consensus.cc:2243] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:47:56.015406 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4ed6c66a4ce141fbabbee58877689439 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ed6c66a4ce141fbabbee58877689439 already in progress: creating tablet
I20251024 19:47:56.015484 1776 raft_consensus.cc:2272] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.016258 1776 ts_tablet_manager.cc:1916] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.021374 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 44d777147a60462e86b3abf18a40f2c9 with delete_type TABLET_DATA_DELETED (replaced by tablet c70e51a4bd7c423aae7d6e2ddf0c60b4) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.021646 1774 tablet_replica.cc:333] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:56.021844 1774 raft_consensus.cc:2243] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.021996 1774 raft_consensus.cc:2272] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.022873 1766 tablet_bootstrap.cc:492] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.022933 1776 ts_tablet_manager.cc:1929] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.023061 1766 ts_tablet_manager.cc:1403] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.029s
I20251024 19:47:56.023087 1776 log.cc:1199] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/5386938a8faf40e0ba1c7838bfbc35bc
I20251024 19:47:56.023471 1776 ts_tablet_manager.cc:1950] T 5386938a8faf40e0ba1c7838bfbc35bc P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.024145 1766 raft_consensus.cc:359] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.024392 1766 raft_consensus.cc:385] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.024611 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 5386938a8faf40e0ba1c7838bfbc35bc with delete_type TABLET_DATA_DELETED (replaced by tablet 07d66fe64bff4a00a15554f212456698) from {username='slave'} at 127.0.0.1:46462
W20251024 19:47:56.025270 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 5386938a8faf40e0ba1c7838bfbc35bc because the tablet was not found. No further retry: Not found: Tablet not found: 5386938a8faf40e0ba1c7838bfbc35bc
I20251024 19:47:56.025318 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 2218f5fe8f77443992a9b83c694e2d20 with delete_type TABLET_DATA_DELETED (replaced by tablet 9af5d5bd80a14f2ebdc7a595b17416e6) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.025449 1766 raft_consensus.cc:740] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.025614 1775 tablet_replica.cc:333] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.025786 1775 raft_consensus.cc:2243] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.025986 1775 raft_consensus.cc:2272] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.026674 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 5386938a8faf40e0ba1c7838bfbc35bc (table default.Table_7c34d3a8affd4e69a31b276ccbe096ae [id=aadca5a1b6d4494aa37f8a8cc76ad12a]) successfully deleted
I20251024 19:47:56.026697 1775 ts_tablet_manager.cc:1916] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.027004 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet ad673e92efc94d33ad07a02b6464e724 with delete_type TABLET_DATA_DELETED (replaced by tablet 44d777147a60462e86b3abf18a40f2c9) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.027906 1774 ts_tablet_manager.cc:1916] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.029862 1782 raft_consensus.cc:493] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:56.033509 1775 ts_tablet_manager.cc:1929] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.033659 1775 log.cc:1199] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/2218f5fe8f77443992a9b83c694e2d20
I20251024 19:47:56.034127 1775 ts_tablet_manager.cc:1950] T 2218f5fe8f77443992a9b83c694e2d20 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.035509 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 2218f5fe8f77443992a9b83c694e2d20 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]) successfully deleted
I20251024 19:47:56.025770 1766 consensus_queue.cc:260] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.039614 1774 ts_tablet_manager.cc:1929] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.039839 1774 log.cc:1199] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/44d777147a60462e86b3abf18a40f2c9
I20251024 19:47:56.040131 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet f34c1080c5554c66b91c99ea294632b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 20f32e34ce51446d8cc7d4938017e60e) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.040424 1774 ts_tablet_manager.cc:1950] T 44d777147a60462e86b3abf18a40f2c9 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:56.041657 1782 raft_consensus.cc:515] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.041934 1766 ts_tablet_manager.cc:1434] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20251024 19:47:56.042061 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 44d777147a60462e86b3abf18a40f2c9 (table default.Table_7ddfa79850884c62900d2e92bc798c64 [id=750d9803961d4003b052aa142e097a45]) successfully deleted
I20251024 19:47:56.042157 1775 tablet_replica.cc:333] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.042305 1775 raft_consensus.cc:2243] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.042380 1782 leader_election.cc:290] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:47:56.042455 1775 raft_consensus.cc:2272] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.043510 1766 tablet_bootstrap.cc:492] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.043781 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad673e92efc94d33ad07a02b6464e724" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:56.044371 1776 tablet_replica.cc:333] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:56.044624 1776 raft_consensus.cc:2243] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:47:56.044371 1586 leader_election.cc:343] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: ad673e92efc94d33ad07a02b6464e724
I20251024 19:47:56.044814 1767 tablet_bootstrap.cc:492] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
W20251024 19:47:56.044971 1586 leader_election.cc:343] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:56.044996 1767 ts_tablet_manager.cc:1403] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.000s
I20251024 19:47:56.045078 1586 leader_election.cc:304] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:47:56.045228 1766 tablet_bootstrap.cc:654] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.045397 1776 raft_consensus.cc:2272] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.045563 1782 raft_consensus.cc:2706] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopped
I20251024 19:47:56.045938 1767 raft_consensus.cc:359] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.046141 1767 raft_consensus.cc:385] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.046200 1775 ts_tablet_manager.cc:1916] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.046229 1767 raft_consensus.cc:740] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.046481 1767 consensus_queue.cc:260] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.047058 1767 ts_tablet_manager.cc:1434] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251024 19:47:56.047662 1768 tablet_bootstrap.cc:492] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.047816 1768 ts_tablet_manager.cc:1403] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.002s
I20251024 19:47:56.048605 1768 raft_consensus.cc:359] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.048784 1768 raft_consensus.cc:385] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.048870 1768 raft_consensus.cc:740] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.049075 1768 consensus_queue.cc:260] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.049789 1767 tablet_bootstrap.cc:492] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:56.052534 1775 ts_tablet_manager.cc:1929] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.052706 1775 log.cc:1199] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/ad673e92efc94d33ad07a02b6464e724
I20251024 19:47:56.053071 1775 ts_tablet_manager.cc:1950] T ad673e92efc94d33ad07a02b6464e724 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.053400 1776 ts_tablet_manager.cc:1916] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.053593 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 0df102c2be684823a8eb3b9c5d40d744 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.053675 1767 tablet_bootstrap.cc:654] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.054137 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet ad673e92efc94d33ad07a02b6464e724 (table default.Table_7ddfa79850884c62900d2e92bc798c64 [id=750d9803961d4003b052aa142e097a45]) successfully deleted
W20251024 19:47:56.054411 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 0df102c2be684823a8eb3b9c5d40d744 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0df102c2be684823a8eb3b9c5d40d744 already in progress: creating tablet
I20251024 19:47:56.055200 1768 ts_tablet_manager.cc:1434] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251024 19:47:56.058264 1768 tablet_bootstrap.cc:492] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.061825 1768 tablet_bootstrap.cc:654] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.063071 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 0df102c2be684823a8eb3b9c5d40d744 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.063341 1774 tablet_replica.cc:333] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:56.063227 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_cc2b5745871f496da3404c94817c9fda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.063519 1774 raft_consensus.cc:2243] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.063733 1774 raft_consensus.cc:2272] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:47:56.063874 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc2b5745871f496da3404c94817c9fda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.064464 1774 ts_tablet_manager.cc:1916] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.065183 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ebef5e3474bb4a288fcbaf6def6a7ce1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.065490 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet af2cc8c940b94b0ab10bfc98c8618e6f
W20251024 19:47:56.065728 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebef5e3474bb4a288fcbaf6def6a7ce1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.066128 1191 catalog_manager.cc:6719] ReplaceTablet: tablet af2cc8c940b94b0ab10bfc98c8618e6f deleted and replaced by tablet 608696ec90ed45e1ad939ed2bfc88d34
I20251024 19:47:56.067222 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet af2cc8c940b94b0ab10bfc98c8618e6f with delete_type TABLET_DATA_DELETED (replaced by tablet 608696ec90ed45e1ad939ed2bfc88d34) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.068570 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9af5d5bd80a14f2ebdc7a595b17416e6
I20251024 19:47:56.069458 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3b0539e760b4408db86d44115e227a8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.070005 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b0539e760b4408db86d44115e227a8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.070798 1774 ts_tablet_manager.cc:1929] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.070827 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet af2cc8c940b94b0ab10bfc98c8618e6f with delete_type TABLET_DATA_DELETED (replaced by tablet 608696ec90ed45e1ad939ed2bfc88d34) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.070914 1774 log.cc:1199] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/0df102c2be684823a8eb3b9c5d40d744
I20251024 19:47:56.071110 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 9af5d5bd80a14f2ebdc7a595b17416e6 with delete_type TABLET_DATA_DELETED (replaced by tablet fb08e56db59b4db796c50db5bda93af9) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.071322 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet af2cc8c940b94b0ab10bfc98c8618e6f with delete_type TABLET_DATA_DELETED (replaced by tablet 608696ec90ed45e1ad939ed2bfc88d34) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.071492 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 9af5d5bd80a14f2ebdc7a595b17416e6 with delete_type TABLET_DATA_DELETED (replaced by tablet fb08e56db59b4db796c50db5bda93af9) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.071599 1774 ts_tablet_manager.cc:1950] T 0df102c2be684823a8eb3b9c5d40d744 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:56.071885 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_59fbd07181e04f8495abd07bda1271c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.072362 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59fbd07181e04f8495abd07bda1271c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.072793 1774 tablet_replica.cc:333] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:56.073016 1774 raft_consensus.cc:2243] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.073185 1774 raft_consensus.cc:2272] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.074131 1774 ts_tablet_manager.cc:1916] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.074383 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet 3cf7d13f588543b1973364fb7e801ffa from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.074998 1776 ts_tablet_manager.cc:1929] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.075980 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_dbcb50e578964f0f9c6d722e4ff4e6a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.076500 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbcb50e578964f0f9c6d722e4ff4e6a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.077595 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 3cf7d13f588543b1973364fb7e801ffa from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.077646 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 0df102c2be684823a8eb3b9c5d40d744 (table default.Table_ae01a61e72ea4e9592e345869e67e586 [id=41f77affc9b0465d8159bbbc2d16646c]) successfully deleted
I20251024 19:47:56.079520 1183 catalog_manager.cc:6719] ReplaceTablet: tablet 9af5d5bd80a14f2ebdc7a595b17416e6 deleted and replaced by tablet fb08e56db59b4db796c50db5bda93af9
I20251024 19:47:56.080449 1774 ts_tablet_manager.cc:1929] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251024 19:47:56.080518 1559 tablet.cc:2378] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 19:47:56.080627 1774 log.cc:1199] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/af2cc8c940b94b0ab10bfc98c8618e6f
I20251024 19:47:56.081048 1774 ts_tablet_manager.cc:1950] T af2cc8c940b94b0ab10bfc98c8618e6f P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:56.082355 1776 log.cc:1199] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/ad673e92efc94d33ad07a02b6464e724
I20251024 19:47:56.082456 1775 tablet_replica.cc:333] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.082614 1775 raft_consensus.cc:2243] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.082805 1776 ts_tablet_manager.cc:1950] T ad673e92efc94d33ad07a02b6464e724 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.082814 1775 raft_consensus.cc:2272] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.084005 1776 tablet_replica.cc:333] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:56.084124 1766 tablet_bootstrap.cc:492] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.084196 1776 raft_consensus.cc:2243] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.084265 1766 ts_tablet_manager.cc:1403] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.000s
I20251024 19:47:56.084357 1776 raft_consensus.cc:2272] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.085005 1766 raft_consensus.cc:359] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.085180 1766 raft_consensus.cc:385] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.085235 1766 raft_consensus.cc:740] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.085472 1766 consensus_queue.cc:260] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.078328 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet 9af5d5bd80a14f2ebdc7a595b17416e6 with delete_type TABLET_DATA_DELETED (replaced by tablet fb08e56db59b4db796c50db5bda93af9) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.081745 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 0df102c2be684823a8eb3b9c5d40d744 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.086853 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet ad673e92efc94d33ad07a02b6464e724 (table default.Table_7ddfa79850884c62900d2e92bc798c64 [id=750d9803961d4003b052aa142e097a45]) successfully deleted
W20251024 19:47:56.087145 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7f9a6a1dc46744a385cdb484524c5c59 because the tablet was not found. No further retry: Not found: Tablet not found: 7f9a6a1dc46744a385cdb484524c5c59
W20251024 19:47:56.087368 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4ed6c66a4ce141fbabbee58877689439 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ed6c66a4ce141fbabbee58877689439 already in progress: creating tablet
I20251024 19:47:56.088240 1779 raft_consensus.cc:493] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:56.088423 1779 raft_consensus.cc:515] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.090456 1779 leader_election.cc:290] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:47:56.092013 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_401da5a2f0dc42bebcb8f9076d40a8b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.092428 1776 ts_tablet_manager.cc:1916] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:47:56.092551 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_401da5a2f0dc42bebcb8f9076d40a8b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.092777 1768 tablet_bootstrap.cc:492] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.092941 1768 ts_tablet_manager.cc:1403] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.000s
I20251024 19:47:56.093783 1768 raft_consensus.cc:359] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.093986 1768 raft_consensus.cc:385] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.094058 1768 raft_consensus.cc:740] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.094359 1768 consensus_queue.cc:260] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.097553 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 26d6f1105b324afba36081c760930ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.097855 1768 ts_tablet_manager.cc:1434] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:47:56.100684 1768 tablet_bootstrap.cc:492] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.101788 1767 tablet_bootstrap.cc:492] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:56.101998 1767 ts_tablet_manager.cc:1403] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.010s
I20251024 19:47:56.102593 1768 tablet_bootstrap.cc:654] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.106156 1774 tablet_replica.cc:333] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:56.106325 1774 raft_consensus.cc:2243] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.106516 1774 raft_consensus.cc:2272] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.109218 1774 ts_tablet_manager.cc:1916] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.113605 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3cf7d13f588543b1973364fb7e801ffa
I20251024 19:47:56.114107 1190 catalog_manager.cc:6719] ReplaceTablet: tablet 3cf7d13f588543b1973364fb7e801ffa deleted and replaced by tablet 1745bbc68d6846fab1a6197f898754b4
I20251024 19:47:56.114758 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet af2cc8c940b94b0ab10bfc98c8618e6f (table default.Table_6335db260a264179aef9dbf353382a6d [id=2c92cb09b287410c90b811e4b88c2260]) successfully deleted
I20251024 19:47:56.115734 1774 ts_tablet_manager.cc:1929] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.115856 1774 log.cc:1199] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/9af5d5bd80a14f2ebdc7a595b17416e6
I20251024 19:47:56.116250 1774 ts_tablet_manager.cc:1950] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:56.117166 1774 tablet_replica.cc:333] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:56.117357 1774 raft_consensus.cc:2243] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.119243 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 922f975a672746b3b72acf4d3050f688 (DEFAULT_TABLE table=default.Table_88ac0b5d20a946f9948436719299d223 [id=eab377d5ce1343d7a12c3740fe46beb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.119483 1351 tablet_service.cc:1505] Processing CreateTablet for tablet ae86e74f810c4b989ac633f59e6ac445 (DEFAULT_TABLE table=default.Table_01b0d0ffefb744f3a1abe0eeab079dc2 [id=071606963978424489f30719fa42baa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.119624 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 922f975a672746b3b72acf4d3050f688. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.119768 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 8b2f85cccb274d9da74bb37f91f020af (DEFAULT_TABLE table=default.Table_8d9e77c4fcbf4a6e9f90d1c111a8cb50 [id=9b044de638ad4f14b886d199bc4a2532]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.120247 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 0ffaa4997edd4b1187579958037ea793 (DEFAULT_TABLE table=default.Table_cf69470650f643c195eb5fbecbd2d2c4 [id=c1c8743e95d34655b1a8779920ed837a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.120884 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 889bd73511b945b4a4e751f60929a5dc (DEFAULT_TABLE table=default.Table_a2fc8ac42bbc44af8bf194f2d3511bd5 [id=5518afdf09da4433bee8f2221ead84c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.121745 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf7d13f588543b1973364fb7e801ffa with delete_type TABLET_DATA_DELETED (replaced by tablet 1745bbc68d6846fab1a6197f898754b4) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.121937 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 922f975a672746b3b72acf4d3050f688 (DEFAULT_TABLE table=default.Table_88ac0b5d20a946f9948436719299d223 [id=eab377d5ce1343d7a12c3740fe46beb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.122344 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 922f975a672746b3b72acf4d3050f688. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.122511 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 8b2f85cccb274d9da74bb37f91f020af (DEFAULT_TABLE table=default.Table_8d9e77c4fcbf4a6e9f90d1c111a8cb50 [id=9b044de638ad4f14b886d199bc4a2532]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.122982 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b2f85cccb274d9da74bb37f91f020af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.117530 1774 raft_consensus.cc:2272] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.125861 1490 tablet_service.cc:1505] Processing CreateTablet for tablet ae86e74f810c4b989ac633f59e6ac445 (DEFAULT_TABLE table=default.Table_01b0d0ffefb744f3a1abe0eeab079dc2 [id=071606963978424489f30719fa42baa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.126049 1774 ts_tablet_manager.cc:1916] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.126122 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 889bd73511b945b4a4e751f60929a5dc (DEFAULT_TABLE table=default.Table_a2fc8ac42bbc44af8bf194f2d3511bd5 [id=5518afdf09da4433bee8f2221ead84c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.126268 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae86e74f810c4b989ac633f59e6ac445. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.126533 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 889bd73511b945b4a4e751f60929a5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.125883 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 0ffaa4997edd4b1187579958037ea793 (DEFAULT_TABLE table=default.Table_cf69470650f643c195eb5fbecbd2d2c4 [id=c1c8743e95d34655b1a8779920ed837a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.126978 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ffaa4997edd4b1187579958037ea793. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.120270 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf7d13f588543b1973364fb7e801ffa with delete_type TABLET_DATA_DELETED (replaced by tablet 1745bbc68d6846fab1a6197f898754b4) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.119858 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae86e74f810c4b989ac633f59e6ac445. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.130021 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ffaa4997edd4b1187579958037ea793. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.130306 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b2f85cccb274d9da74bb37f91f020af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.130532 1768 tablet_bootstrap.cc:492] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.130686 1768 ts_tablet_manager.cc:1403] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.004s
I20251024 19:47:56.131271 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 889bd73511b945b4a4e751f60929a5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.131637 1768 raft_consensus.cc:359] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.131918 1768 raft_consensus.cc:385] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.132031 1768 raft_consensus.cc:740] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.132848 1768 consensus_queue.cc:260] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.133562 1768 ts_tablet_manager.cc:1434] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251024 19:47:56.133915 1768 tablet_bootstrap.cc:492] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.135936 1768 tablet_bootstrap.cc:654] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.138069 1768 tablet_bootstrap.cc:492] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.138262 1768 ts_tablet_manager.cc:1403] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251024 19:47:56.139086 1768 raft_consensus.cc:359] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.139271 1768 raft_consensus.cc:385] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.139344 1768 raft_consensus.cc:740] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.139635 1768 consensus_queue.cc:260] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.140129 1768 ts_tablet_manager.cc:1434] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:47:56.140501 1768 tablet_bootstrap.cc:492] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.141917 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6335db260a264179aef9dbf353382a6d" } new_table_name: "default.Table_6e1fbc007b464e8c93803ba262a19fea" modify_external_catalogs: true
I20251024 19:47:56.142261 1768 tablet_bootstrap.cc:654] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.142408 1192 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet b96722ee06cf4e26a55a913c060a1209 (table default.Table_4dda32c7ff144630bebd572cf5f9f822 [id=350b37bf42194fcfbb660bffe83e47c4]). Expected version 1 got 0
I20251024 19:47:56.143909 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 9af5d5bd80a14f2ebdc7a595b17416e6 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]) successfully deleted
I20251024 19:47:56.144639 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_85e6c4c107c94e7abfc7c3396aaf6e81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.144898 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_95644b1e2fa0449d806efb9c458266b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.145128 1184 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet b96722ee06cf4e26a55a913c060a1209 (table default.Table_4dda32c7ff144630bebd572cf5f9f822 [id=350b37bf42194fcfbb660bffe83e47c4]). Expected version 1 got 0
W20251024 19:47:56.145247 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85e6c4c107c94e7abfc7c3396aaf6e81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:56.145488 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95644b1e2fa0449d806efb9c458266b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.147104 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 20f32e34ce51446d8cc7d4938017e60e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.147485 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3b0539e760b4408db86d44115e227a8d" } new_table_name: "default.Table_59f512ddad6842eca4865d6387dee631" modify_external_catalogs: true
I20251024 19:47:56.148715 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet 7e31eef151884385854096ff71484a9c from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.148981 1192 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 7e31eef151884385854096ff71484a9c (table default.Table_dabc8d67782e4372b14bcf1862b25d3a [id=4e6695ccb7d342a6985b2a10a6bdfaac]). Expected version 2 got 0
I20251024 19:47:56.149048 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8ca12a13ba4742478879c608bd9ddf3e" } new_table_name: "default.Table_ee4ef17d5f314379846d30a477d4911d" modify_external_catalogs: true
I20251024 19:47:56.156674 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a74b3cfcbf75476cabf73a2e79f6df92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.157229 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a74b3cfcbf75476cabf73a2e79f6df92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:56.158278 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 608696ec90ed45e1ad939ed2bfc88d34 has no consensus state
W20251024 19:47:56.162370 1447 leader_election.cc:343] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: af2cc8c940b94b0ab10bfc98c8618e6f
W20251024 19:47:56.169150 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d941eafb1414f068dc656946082f617 consensus state has no leader
W20251024 19:47:56.169482 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0397637f0778442a871528d5dd173689 has no consensus state
I20251024 19:47:56.169865 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7e31eef151884385854096ff71484a9c
I20251024 19:47:56.170365 1191 catalog_manager.cc:6719] ReplaceTablet: tablet 7e31eef151884385854096ff71484a9c deleted and replaced by tablet 7cbbedcba0f04ea6b00670e921d25200
I20251024 19:47:56.172712 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e31eef151884385854096ff71484a9c with delete_type TABLET_DATA_DELETED (replaced by tablet 7cbbedcba0f04ea6b00670e921d25200) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.173090 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e31eef151884385854096ff71484a9c with delete_type TABLET_DATA_DELETED (replaced by tablet 7cbbedcba0f04ea6b00670e921d25200) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.174583 1766 ts_tablet_manager.cc:1434] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.090s user 0.002s sys 0.000s
I20251024 19:47:56.110996 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 922f975a672746b3b72acf4d3050f688 (DEFAULT_TABLE table=default.Table_88ac0b5d20a946f9948436719299d223 [id=eab377d5ce1343d7a12c3740fe46beb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.177999 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 922f975a672746b3b72acf4d3050f688. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.179352 1766 tablet_bootstrap.cc:492] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.179965 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 2d941eafb1414f068dc656946082f617 (DEFAULT_TABLE table=default.Table_8ca12a13ba4742478879c608bd9ddf3e [id=a402387542654ccf8e0b04ca8ecebb1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.180466 1487 tablet_service.cc:1505] Processing CreateTablet for tablet fb17a35d8d8446cf9870d05e107ed237 (DEFAULT_TABLE table=default.Table_a35e66c1df02471480ce2e87424f6e91 [id=9a05bcf7ad5d430c9ef6d6e18a27808e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.181360 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d941eafb1414f068dc656946082f617. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.111375 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 8b2f85cccb274d9da74bb37f91f020af (DEFAULT_TABLE table=default.Table_8d9e77c4fcbf4a6e9f90d1c111a8cb50 [id=9b044de638ad4f14b886d199bc4a2532]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.180923 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 2d941eafb1414f068dc656946082f617 (DEFAULT_TABLE table=default.Table_8ca12a13ba4742478879c608bd9ddf3e [id=a402387542654ccf8e0b04ca8ecebb1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.181900 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b2f85cccb274d9da74bb37f91f020af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.182269 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d941eafb1414f068dc656946082f617. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.182266 1351 tablet_service.cc:1505] Processing CreateTablet for tablet fb17a35d8d8446cf9870d05e107ed237 (DEFAULT_TABLE table=default.Table_a35e66c1df02471480ce2e87424f6e91 [id=9a05bcf7ad5d430c9ef6d6e18a27808e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.182778 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb17a35d8d8446cf9870d05e107ed237. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.117714 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 0ffaa4997edd4b1187579958037ea793 (DEFAULT_TABLE table=default.Table_cf69470650f643c195eb5fbecbd2d2c4 [id=c1c8743e95d34655b1a8779920ed837a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.185775 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb17a35d8d8446cf9870d05e107ed237. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.118129 1625 tablet_service.cc:1505] Processing CreateTablet for tablet ae86e74f810c4b989ac633f59e6ac445 (DEFAULT_TABLE table=default.Table_01b0d0ffefb744f3a1abe0eeab079dc2 [id=071606963978424489f30719fa42baa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.118353 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf7d13f588543b1973364fb7e801ffa with delete_type TABLET_DATA_DELETED (replaced by tablet 1745bbc68d6846fab1a6197f898754b4) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.131191 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af2cc8c940b94b0ab10bfc98c8618e6f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:56.132608 1776 ts_tablet_manager.cc:1929] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.163425 1627 tablet_service.cc:1505] Processing CreateTablet for tablet 889bd73511b945b4a4e751f60929a5dc (DEFAULT_TABLE table=default.Table_a2fc8ac42bbc44af8bf194f2d3511bd5 [id=5518afdf09da4433bee8f2221ead84c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.164147 1626 tablet_service.cc:1505] Processing CreateTablet for tablet 2d941eafb1414f068dc656946082f617 (DEFAULT_TABLE table=default.Table_8ca12a13ba4742478879c608bd9ddf3e [id=a402387542654ccf8e0b04ca8ecebb1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.165011 1623 tablet_service.cc:1505] Processing CreateTablet for tablet fb17a35d8d8446cf9870d05e107ed237 (DEFAULT_TABLE table=default.Table_a35e66c1df02471480ce2e87424f6e91 [id=9a05bcf7ad5d430c9ef6d6e18a27808e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.171340 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e31eef151884385854096ff71484a9c with delete_type TABLET_DATA_DELETED (replaced by tablet 7cbbedcba0f04ea6b00670e921d25200) from {username='slave'} at 127.0.0.1:54246
W20251024 19:47:56.188175 1444 leader_election.cc:343] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: Tablet not RUNNING: STOPPED
I20251024 19:47:56.188337 1444 leader_election.cc:304] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:47:56.189003 1775 ts_tablet_manager.cc:1916] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.189116 1779 raft_consensus.cc:2749] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:56.191021 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ffaa4997edd4b1187579958037ea793. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.191650 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d941eafb1414f068dc656946082f617. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.191890 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 889bd73511b945b4a4e751f60929a5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.192065 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae86e74f810c4b989ac633f59e6ac445. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.192607 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb17a35d8d8446cf9870d05e107ed237. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.181840 1766 tablet_bootstrap.cc:654] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.193608 1776 log.cc:1199] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/af2cc8c940b94b0ab10bfc98c8618e6f
I20251024 19:47:56.193991 1776 ts_tablet_manager.cc:1950] T af2cc8c940b94b0ab10bfc98c8618e6f P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.195201 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet af2cc8c940b94b0ab10bfc98c8618e6f (table default.Table_6e1fbc007b464e8c93803ba262a19fea [id=2c92cb09b287410c90b811e4b88c2260]) successfully deleted
I20251024 19:47:56.195689 1621 tablet_service.cc:1505] Processing CreateTablet for tablet fc8c01a4c07a42a0bc48a58313090e68 (DEFAULT_TABLE table=default.Table_401da5a2f0dc42bebcb8f9076d40a8b8 [id=d955ab5050fa4039b48b0c9235b687cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.195932 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 97c6dce647064344b0b53a7ffaeafb75 (DEFAULT_TABLE table=default.Table_85e6c4c107c94e7abfc7c3396aaf6e81 [id=1b078c2814dd4ff99c1e2ee01d0159cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.196300 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc8c01a4c07a42a0bc48a58313090e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.196516 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97c6dce647064344b0b53a7ffaeafb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.196671 1624 tablet_service.cc:1505] Processing CreateTablet for tablet e2d97caee586418a8981108465f6b704 (DEFAULT_TABLE table=default.Table_95644b1e2fa0449d806efb9c458266b9 [id=168a2adf03184a04b7e8f5017919a08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.197387 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2d97caee586418a8981108465f6b704. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.198239 1776 tablet_replica.cc:333] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:47:56.198490 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9af5d5bd80a14f2ebdc7a595b17416e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9af5d5bd80a14f2ebdc7a595b17416e6 already in progress: creating tablet
I20251024 19:47:56.198992 1776 raft_consensus.cc:2243] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.199131 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 1745bbc68d6846fab1a6197f898754b4 (DEFAULT_TABLE table=default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.199213 1776 raft_consensus.cc:2272] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.200224 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 97c6dce647064344b0b53a7ffaeafb75 (DEFAULT_TABLE table=default.Table_85e6c4c107c94e7abfc7c3396aaf6e81 [id=1b078c2814dd4ff99c1e2ee01d0159cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.200719 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97c6dce647064344b0b53a7ffaeafb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.201074 1776 ts_tablet_manager.cc:1916] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.201443 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 608696ec90ed45e1ad939ed2bfc88d34 (DEFAULT_TABLE table=default.Table_6e1fbc007b464e8c93803ba262a19fea [id=2c92cb09b287410c90b811e4b88c2260]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.201889 1350 tablet_service.cc:1505] Processing CreateTablet for tablet e2d97caee586418a8981108465f6b704 (DEFAULT_TABLE table=default.Table_95644b1e2fa0449d806efb9c458266b9 [id=168a2adf03184a04b7e8f5017919a08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.202291 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2d97caee586418a8981108465f6b704. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.202627 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_88ac0b5d20a946f9948436719299d223" } new_table_name: "default.Table_d98953b765964ecdb15163f3d5e01797" modify_external_catalogs: true
I20251024 19:47:56.202912 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_401da5a2f0dc42bebcb8f9076d40a8b8" } modify_external_catalogs: true
I20251024 19:47:56.203310 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_401da5a2f0dc42bebcb8f9076d40a8b8" } modify_external_catalogs: true
I20251024 19:47:56.203929 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dabc8d67782e4372b14bcf1862b25d3a" } modify_external_catalogs: true
I20251024 19:47:56.204094 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dabc8d67782e4372b14bcf1862b25d3a" } modify_external_catalogs: true
I20251024 19:47:56.204442 1488 tablet_service.cc:1505] Processing CreateTablet for tablet fc8c01a4c07a42a0bc48a58313090e68 (DEFAULT_TABLE table=default.Table_401da5a2f0dc42bebcb8f9076d40a8b8 [id=d955ab5050fa4039b48b0c9235b687cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.204651 1490 tablet_service.cc:1505] Processing CreateTablet for tablet e2d97caee586418a8981108465f6b704 (DEFAULT_TABLE table=default.Table_95644b1e2fa0449d806efb9c458266b9 [id=168a2adf03184a04b7e8f5017919a08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.204818 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc8c01a4c07a42a0bc48a58313090e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.205173 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 1745bbc68d6846fab1a6197f898754b4 (DEFAULT_TABLE table=default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.205747 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1745bbc68d6846fab1a6197f898754b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.206005 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 047436af5f6d4f688bd83e33089d3ee4 (DEFAULT_TABLE table=default.Table_cc2b5745871f496da3404c94817c9fda [id=93f77b3367c04c9bb7e3e02cfa050c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.205237 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 047436af5f6d4f688bd83e33089d3ee4 (DEFAULT_TABLE table=default.Table_cc2b5745871f496da3404c94817c9fda [id=93f77b3367c04c9bb7e3e02cfa050c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.206540 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 047436af5f6d4f688bd83e33089d3ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.209712 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2d97caee586418a8981108465f6b704. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.209975 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 608696ec90ed45e1ad939ed2bfc88d34 (DEFAULT_TABLE table=default.Table_6e1fbc007b464e8c93803ba262a19fea [id=2c92cb09b287410c90b811e4b88c2260]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.210501 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 608696ec90ed45e1ad939ed2bfc88d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.214067 1483 tablet_service.cc:1505] Processing CreateTablet for tablet fb08e56db59b4db796c50db5bda93af9 (DEFAULT_TABLE table=default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.214212 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a74b3cfcbf75476cabf73a2e79f6df92" } modify_external_catalogs: true
I20251024 19:47:56.214390 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a74b3cfcbf75476cabf73a2e79f6df92" } modify_external_catalogs: true
I20251024 19:47:56.214681 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb08e56db59b4db796c50db5bda93af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.215826 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_95644b1e2fa0449d806efb9c458266b9" } modify_external_catalogs: true
I20251024 19:47:56.215977 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_95644b1e2fa0449d806efb9c458266b9" } modify_external_catalogs: true
I20251024 19:47:56.217440 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 8ba85164015448d4a688a473cce90c5c (DEFAULT_TABLE table=default.Table_dbcb50e578964f0f9c6d722e4ff4e6a4 [id=0fa321e2d4d847e69435f6de429d8779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.217969 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1745bbc68d6846fab1a6197f898754b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.218067 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ba85164015448d4a688a473cce90c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.219374 1617 tablet_service.cc:1505] Processing CreateTablet for tablet fb08e56db59b4db796c50db5bda93af9 (DEFAULT_TABLE table=default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.219638 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 047436af5f6d4f688bd83e33089d3ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.220439 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8d9e77c4fcbf4a6e9f90d1c111a8cb50" } modify_external_catalogs: true
I20251024 19:47:56.220587 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8d9e77c4fcbf4a6e9f90d1c111a8cb50" } modify_external_catalogs: true
I20251024 19:47:56.202378 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cc2b5745871f496da3404c94817c9fda" } new_table_name: "default.Table_812ae4fbc0fc4fdea1317e0d9794c50b" modify_external_catalogs: true
I20251024 19:47:56.224993 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 608696ec90ed45e1ad939ed2bfc88d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.228296 1774 ts_tablet_manager.cc:1929] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.231889 1779 raft_consensus.cc:493] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:56.232053 1779 raft_consensus.cc:515] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.232775 1779 leader_election.cc:290] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:47:56.235934 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 1745bbc68d6846fab1a6197f898754b4 (DEFAULT_TABLE table=default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.235934 1347 tablet_service.cc:1505] Processing CreateTablet for tablet fc8c01a4c07a42a0bc48a58313090e68 (DEFAULT_TABLE table=default.Table_401da5a2f0dc42bebcb8f9076d40a8b8 [id=d955ab5050fa4039b48b0c9235b687cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.238765 1775 ts_tablet_manager.cc:1929] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.238893 1775 log.cc:1199] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/f34c1080c5554c66b91c99ea294632b4
I20251024 19:47:56.239274 1775 ts_tablet_manager.cc:1950] T f34c1080c5554c66b91c99ea294632b4 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.240232 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet f34c1080c5554c66b91c99ea294632b4 (table default.Table_835ab1cc0a82456f8282bce320dca4f8 [id=91d53300d972437d9efba6385dbef72c]) successfully deleted
I20251024 19:47:56.241161 1766 tablet_bootstrap.cc:492] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.241331 1766 ts_tablet_manager.cc:1403] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.062s user 0.000s sys 0.004s
I20251024 19:47:56.241683 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 047436af5f6d4f688bd83e33089d3ee4 (DEFAULT_TABLE table=default.Table_cc2b5745871f496da3404c94817c9fda [id=93f77b3367c04c9bb7e3e02cfa050c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.241861 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 97c6dce647064344b0b53a7ffaeafb75 (DEFAULT_TABLE table=default.Table_85e6c4c107c94e7abfc7c3396aaf6e81 [id=1b078c2814dd4ff99c1e2ee01d0159cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.242358 1766 raft_consensus.cc:359] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.242606 1766 raft_consensus.cc:385] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.242729 1766 raft_consensus.cc:740] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.243005 1766 consensus_queue.cc:260] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:47:56.244156 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 922f975a672746b3b72acf4d3050f688 consensus state has no leader
I20251024 19:47:56.249048 1766 ts_tablet_manager.cc:1434] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20251024 19:47:56.249195 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 8ba85164015448d4a688a473cce90c5c (DEFAULT_TABLE table=default.Table_dbcb50e578964f0f9c6d722e4ff4e6a4 [id=0fa321e2d4d847e69435f6de429d8779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.249951 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 0397637f0778442a871528d5dd173689 (DEFAULT_TABLE table=default.Table_59f512ddad6842eca4865d6387dee631 [id=e6d5a0f628404ad0bd19cc019911e269]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.250427 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0397637f0778442a871528d5dd173689. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.255498 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 0397637f0778442a871528d5dd173689 (DEFAULT_TABLE table=default.Table_59f512ddad6842eca4865d6387dee631 [id=e6d5a0f628404ad0bd19cc019911e269]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.256048 1189 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 7cbbedcba0f04ea6b00670e921d25200
I20251024 19:47:56.257840 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 8ba85164015448d4a688a473cce90c5c (DEFAULT_TABLE table=default.Table_dbcb50e578964f0f9c6d722e4ff4e6a4 [id=0fa321e2d4d847e69435f6de429d8779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.259819 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 0397637f0778442a871528d5dd173689 (DEFAULT_TABLE table=default.Table_59f512ddad6842eca4865d6387dee631 [id=e6d5a0f628404ad0bd19cc019911e269]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.259881 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c (DEFAULT_TABLE table=default.Table_6b2b5091b93348958149ec3640ed6971 [id=a1c4569aa6934d4a99787cd9e3240edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.260435 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.260772 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c (DEFAULT_TABLE table=default.Table_6b2b5091b93348958149ec3640ed6971 [id=a1c4569aa6934d4a99787cd9e3240edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.263024 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 65f646bd05274cb6859c0b7f92b61352 (DEFAULT_TABLE table=default.Table_59fbd07181e04f8495abd07bda1271c0 [id=665389c7bde54b7786edabcb91a9b33b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.263684 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet fc8c01a4c07a42a0bc48a58313090e68
I20251024 19:47:56.265626 1766 tablet_bootstrap.cc:492] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.267788 1766 tablet_bootstrap.cc:654] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.267935 1768 tablet_bootstrap.cc:492] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.268108 1768 ts_tablet_manager.cc:1403] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.128s user 0.017s sys 0.008s
I20251024 19:47:56.268919 1768 raft_consensus.cc:359] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.269120 1768 raft_consensus.cc:385] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.269212 1768 raft_consensus.cc:740] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.269459 1768 consensus_queue.cc:260] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.270607 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 230606a0ce084b7bab0e41e48cf8fa89 (DEFAULT_TABLE table=default.Table_ebef5e3474bb4a288fcbaf6def6a7ce1 [id=469c675d15e3482d98671ee6b9e4bd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.271085 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 230606a0ce084b7bab0e41e48cf8fa89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.272480 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 65f646bd05274cb6859c0b7f92b61352 (DEFAULT_TABLE table=default.Table_59fbd07181e04f8495abd07bda1271c0 [id=665389c7bde54b7786edabcb91a9b33b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.272938 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f646bd05274cb6859c0b7f92b61352. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.273463 1185 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 25db457af955403f83d9e4b1a16e69b4
I20251024 19:47:56.276340 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 230606a0ce084b7bab0e41e48cf8fa89 (DEFAULT_TABLE table=default.Table_ebef5e3474bb4a288fcbaf6def6a7ce1 [id=469c675d15e3482d98671ee6b9e4bd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.277267 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 0df102c2be684823a8eb3b9c5d40d744 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.276777 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 230606a0ce084b7bab0e41e48cf8fa89 (DEFAULT_TABLE table=default.Table_ebef5e3474bb4a288fcbaf6def6a7ce1 [id=469c675d15e3482d98671ee6b9e4bd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.279361 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet fc8c01a4c07a42a0bc48a58313090e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.280195 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet fc8c01a4c07a42a0bc48a58313090e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.280725 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c (DEFAULT_TABLE table=default.Table_6b2b5091b93348958149ec3640ed6971 [id=a1c4569aa6934d4a99787cd9e3240edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.281137 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet fc8c01a4c07a42a0bc48a58313090e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.281433 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet e2d97caee586418a8981108465f6b704
I20251024 19:47:56.278743 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 65f646bd05274cb6859c0b7f92b61352 (DEFAULT_TABLE table=default.Table_59fbd07181e04f8495abd07bda1271c0 [id=665389c7bde54b7786edabcb91a9b33b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.284471 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet e2d97caee586418a8981108465f6b704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.285496 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9a372068f5b34bdcbe40f464da2797a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.286109 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a372068f5b34bdcbe40f464da2797a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.286386 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet e2d97caee586418a8981108465f6b704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.286564 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet e2d97caee586418a8981108465f6b704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.287482 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2cbb0aad326e4bae85d4c5cb833150c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.287984 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cbb0aad326e4bae85d4c5cb833150c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.288677 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3d57d480ef4c4a1492438c1b7dd6135c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.289152 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d57d480ef4c4a1492438c1b7dd6135c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.289290 1775 tablet_replica.cc:333] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.289543 1775 raft_consensus.cc:2243] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.289764 1775 raft_consensus.cc:2272] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.290340 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_be4f56e3a9054dff8204c38760c34f8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.290794 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be4f56e3a9054dff8204c38760c34f8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.296514 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 8b2f85cccb274d9da74bb37f91f020af
I20251024 19:47:56.297775 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b2f85cccb274d9da74bb37f91f020af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.298285 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b2f85cccb274d9da74bb37f91f020af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.296241 1766 tablet_bootstrap.cc:492] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.299376 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b2f85cccb274d9da74bb37f91f020af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.299501 1766 ts_tablet_manager.cc:1403] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.003s
I20251024 19:47:56.300570 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e31eef151884385854096ff71484a9c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:56.304044 1775 ts_tablet_manager.cc:1916] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.304812 1768 ts_tablet_manager.cc:1434] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20251024 19:47:56.306015 1766 raft_consensus.cc:359] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.307196 1766 raft_consensus.cc:385] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.307679 1768 tablet_bootstrap.cc:492] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.307803 1766 raft_consensus.cc:740] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
W20251024 19:47:56.309684 1444 leader_election.cc:343] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:56.309945 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 74c8c9949988403890a96ca535d367aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.309175 1766 consensus_queue.cc:260] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:47:56.311837 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 047436af5f6d4f688bd83e33089d3ee4 consensus state has no leader
I20251024 19:47:56.312378 1767 raft_consensus.cc:359] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.312580 1767 raft_consensus.cc:385] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.312683 1767 raft_consensus.cc:740] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.312903 1767 consensus_queue.cc:260] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.317070 1766 ts_tablet_manager.cc:1434] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.017s user 0.001s sys 0.002s
I20251024 19:47:56.323572 1766 tablet_bootstrap.cc:492] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.324375 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9291fbfe8e484be094f28ad94814b13f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.324687 1767 ts_tablet_manager.cc:1434] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.120s user 0.000s sys 0.002s
W20251024 19:47:56.324963 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9291fbfe8e484be094f28ad94814b13f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.326296 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_272323b2c4ac4c8cb0ff412a632ee43d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.326432 1767 tablet_bootstrap.cc:492] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:56.327003 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b728c3d6fcf44b4593b7883fcf4ea07d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.330132 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f646bd05274cb6859c0b7f92b61352. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:56.330425 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b728c3d6fcf44b4593b7883fcf4ea07d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.330540 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 9af5d5bd80a14f2ebdc7a595b17416e6 with delete_type TABLET_DATA_DELETED (replaced by tablet fb08e56db59b4db796c50db5bda93af9) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.330811 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc8c01a4c07a42a0bc48a58313090e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.331789 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:56.331769 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_272323b2c4ac4c8cb0ff412a632ee43d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.333659 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 608696ec90ed45e1ad939ed2bfc88d34 (DEFAULT_TABLE table=default.Table_6e1fbc007b464e8c93803ba262a19fea [id=2c92cb09b287410c90b811e4b88c2260]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.333887 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 230606a0ce084b7bab0e41e48cf8fa89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.334259 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 608696ec90ed45e1ad939ed2bfc88d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.335819 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1745bbc68d6846fab1a6197f898754b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.337543 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0397637f0778442a871528d5dd173689. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.338011 1344 tablet_service.cc:1505] Processing CreateTablet for tablet fb08e56db59b4db796c50db5bda93af9 (DEFAULT_TABLE table=default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.338568 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb08e56db59b4db796c50db5bda93af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.339105 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 44d777147a60462e86b3abf18a40f2c9 with delete_type TABLET_DATA_DELETED (replaced by tablet c70e51a4bd7c423aae7d6e2ddf0c60b4) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.339390 1766 tablet_bootstrap.cc:654] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.341190 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.341714 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0397637f0778442a871528d5dd173689. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.341854 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e31eef151884385854096ff71484a9c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:47:56.342231 1371 raft_consensus.cc:2468] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:47:56.343125 1447 leader_election.cc:304] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:47:56.343613 1779 raft_consensus.cc:2804] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:47:56.343744 1779 raft_consensus.cc:493] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:47:56.343837 1779 raft_consensus.cc:3060] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:56.344554 1775 ts_tablet_manager.cc:1929] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.344686 1775 log.cc:1199] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/af2cc8c940b94b0ab10bfc98c8618e6f
I20251024 19:47:56.345057 1775 ts_tablet_manager.cc:1950] T af2cc8c940b94b0ab10bfc98c8618e6f P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.345746 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ba85164015448d4a688a473cce90c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.347658 1766 tablet_bootstrap.cc:492] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.347805 1766 ts_tablet_manager.cc:1403] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20251024 19:47:56.348636 1766 raft_consensus.cc:359] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.348798 1766 raft_consensus.cc:385] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.348891 1766 raft_consensus.cc:740] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.349013 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ba85164015448d4a688a473cce90c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.349148 1766 consensus_queue.cc:260] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.349557 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb08e56db59b4db796c50db5bda93af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.350346 1775 tablet_replica.cc:333] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.351679 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet af2cc8c940b94b0ab10bfc98c8618e6f (table default.Table_6e1fbc007b464e8c93803ba262a19fea [id=2c92cb09b287410c90b811e4b88c2260]) successfully deleted
W20251024 19:47:56.351980 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9af5d5bd80a14f2ebdc7a595b17416e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9af5d5bd80a14f2ebdc7a595b17416e6 already in progress: creating tablet
W20251024 19:47:56.352329 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3cf7d13f588543b1973364fb7e801ffa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cf7d13f588543b1973364fb7e801ffa already in progress: creating tablet
I20251024 19:47:56.352659 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet 2d941eafb1414f068dc656946082f617 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.353302 1774 log.cc:1199] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/26d6f1105b324afba36081c760930ef0
I20251024 19:47:56.354928 1779 raft_consensus.cc:515] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.355772 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97c6dce647064344b0b53a7ffaeafb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.353485 1767 tablet_bootstrap.cc:654] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.357640 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 047436af5f6d4f688bd83e33089d3ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.357209 1766 ts_tablet_manager.cc:1434] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20251024 19:47:56.360080 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e31eef151884385854096ff71484a9c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
W20251024 19:47:56.360625 1444 leader_election.cc:343] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:56.360850 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet b96722ee06cf4e26a55a913c060a1209 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.364142 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_74349eed6d4b4e8e95ae01909bbb91e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.364773 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74349eed6d4b4e8e95ae01909bbb91e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.364970 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d98953b765964ecdb15163f3d5e01797" } new_table_name: "default.Table_6b632546d5924db7afe499fc1b1b9db0" modify_external_catalogs: true
I20251024 19:47:56.366230 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 20f32e34ce51446d8cc7d4938017e60e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.366823 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 230606a0ce084b7bab0e41e48cf8fa89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.367956 1766 tablet_bootstrap.cc:492] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.368299 1774 ts_tablet_manager.cc:1950] T 26d6f1105b324afba36081c760930ef0 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:56.369156 1774 tablet_replica.cc:333] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:56.369814 1766 tablet_bootstrap.cc:654] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.370586 1768 tablet_bootstrap.cc:654] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.371817 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 26d6f1105b324afba36081c760930ef0 (table default.Table_2759324fce09406ab532d22228541f6e [id=30be39f295db49d98ee0c102dd6b3b14]) successfully deleted
I20251024 19:47:56.372037 1774 raft_consensus.cc:2243] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:47:56.372174 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3cf7d13f588543b1973364fb7e801ffa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cf7d13f588543b1973364fb7e801ffa already in progress: creating tablet
I20251024 19:47:56.376695 1774 raft_consensus.cc:2272] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.376924 1779 leader_election.cc:290] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:47:56.377732 1766 tablet_bootstrap.cc:492] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.377970 1766 ts_tablet_manager.cc:1403] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20251024 19:47:56.378698 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f646bd05274cb6859c0b7f92b61352. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.378908 1766 raft_consensus.cc:359] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.379207 1766 raft_consensus.cc:385] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.379335 1766 raft_consensus.cc:740] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.379698 1766 consensus_queue.cc:260] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.379797 1774 ts_tablet_manager.cc:1916] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.380405 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e31eef151884385854096ff71484a9c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:47:56.380621 1371 raft_consensus.cc:3060] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:56.381567 1775 raft_consensus.cc:2243] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.382736 1775 raft_consensus.cc:2272] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.383656 1775 ts_tablet_manager.cc:1916] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.384642 1371 raft_consensus.cc:2468] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:47:56.384878 1776 ts_tablet_manager.cc:1929] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.385074 1776 log.cc:1199] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/0df102c2be684823a8eb3b9c5d40d744
I20251024 19:47:56.385254 1447 leader_election.cc:304] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
W20251024 19:47:56.385507 1447 raft_consensus.cc:2686] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251024 19:47:56.385670 1776 ts_tablet_manager.cc:1950] T 0df102c2be684823a8eb3b9c5d40d744 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.386688 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 0df102c2be684823a8eb3b9c5d40d744 (table default.Table_ae01a61e72ea4e9592e345869e67e586 [id=41f77affc9b0465d8159bbbc2d16646c]) successfully deleted
W20251024 19:47:56.387667 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3cf7d13f588543b1973364fb7e801ffa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cf7d13f588543b1973364fb7e801ffa already in progress: creating tablet
I20251024 19:47:56.388177 1766 ts_tablet_manager.cc:1434] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
W20251024 19:47:56.388823 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet fc8c01a4c07a42a0bc48a58313090e68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc8c01a4c07a42a0bc48a58313090e68 already in progress: creating tablet
W20251024 19:47:56.389081 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet e2d97caee586418a8981108465f6b704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2d97caee586418a8981108465f6b704 already in progress: creating tablet
I20251024 19:47:56.389703 1767 tablet_bootstrap.cc:492] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:56.389930 1767 ts_tablet_manager.cc:1403] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.001s
I20251024 19:47:56.390918 1767 raft_consensus.cc:359] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.391207 1767 raft_consensus.cc:385] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.391335 1767 raft_consensus.cc:740] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.391657 1766 tablet_bootstrap.cc:492] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.391927 1767 consensus_queue.cc:260] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.393347 1766 tablet_bootstrap.cc:654] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.396698 1767 ts_tablet_manager.cc:1434] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251024 19:47:56.397253 1775 ts_tablet_manager.cc:1929] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.397444 1775 log.cc:1199] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/7e31eef151884385854096ff71484a9c
I20251024 19:47:56.398026 1775 ts_tablet_manager.cc:1950] T 7e31eef151884385854096ff71484a9c P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.399953 1767 tablet_bootstrap.cc:492] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:47:56.400627 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 8b2f85cccb274d9da74bb37f91f020af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b2f85cccb274d9da74bb37f91f020af already in progress: creating tablet
I20251024 19:47:56.400774 1775 tablet_replica.cc:333] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.401024 1776 tablet_replica.cc:333] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:56.401372 1776 raft_consensus.cc:2243] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.401955 1775 raft_consensus.cc:2243] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.402309 1775 raft_consensus.cc:2272] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.404138 1776 raft_consensus.cc:2272] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.405818 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c772dc06afa04976b56055df70060bb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.406693 1775 ts_tablet_manager.cc:1916] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:47:56.406750 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c772dc06afa04976b56055df70060bb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.406931 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e31eef151884385854096ff71484a9c with delete_type TABLET_DATA_DELETED (replaced by tablet 7cbbedcba0f04ea6b00670e921d25200) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.408119 1776 ts_tablet_manager.cc:1916] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.409240 1772 raft_consensus.cc:493] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:56.409157 1767 tablet_bootstrap.cc:654] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.409440 1772 raft_consensus.cc:515] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.411093 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b96722ee06cf4e26a55a913c060a1209" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:56.412359 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b96722ee06cf4e26a55a913c060a1209" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:47:56.412856 1308 leader_election.cc:343] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:47:56.413537 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 047436af5f6d4f688bd83e33089d3ee4 (table default.Table_812ae4fbc0fc4fdea1317e0d9794c50b [id=93f77b3367c04c9bb7e3e02cfa050c36], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 047436af5f6d4f688bd83e33089d3ee4 consensus state has no leader
I20251024 19:47:56.414029 1772 leader_election.cc:290] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:47:56.414179 1305 leader_election.cc:343] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:56.415078 1305 leader_election.cc:304] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:56.418673 1774 ts_tablet_manager.cc:1929] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.418929 1774 log.cc:1199] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/20f32e34ce51446d8cc7d4938017e60e
I20251024 19:47:56.419579 1774 ts_tablet_manager.cc:1950] T 20f32e34ce51446d8cc7d4938017e60e P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:56.418677 1772 raft_consensus.cc:2749] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:56.421423 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 20f32e34ce51446d8cc7d4938017e60e (table default.Table_835ab1cc0a82456f8282bce320dca4f8 [id=91d53300d972437d9efba6385dbef72c]) successfully deleted
I20251024 19:47:56.421705 1775 ts_tablet_manager.cc:1929] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.422152 1775 log.cc:1199] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/0df102c2be684823a8eb3b9c5d40d744
I20251024 19:47:56.422868 1775 ts_tablet_manager.cc:1950] T 0df102c2be684823a8eb3b9c5d40d744 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.423369 1774 tablet_replica.cc:333] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:56.424393 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 0df102c2be684823a8eb3b9c5d40d744 (table default.Table_ae01a61e72ea4e9592e345869e67e586 [id=41f77affc9b0465d8159bbbc2d16646c]) successfully deleted
I20251024 19:47:56.424793 1774 raft_consensus.cc:2243] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.425081 1774 raft_consensus.cc:2272] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.425848 1774 ts_tablet_manager.cc:1916] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:47:56.426189 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet fc8c01a4c07a42a0bc48a58313090e68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc8c01a4c07a42a0bc48a58313090e68 already in progress: creating tablet
W20251024 19:47:56.426986 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet e2d97caee586418a8981108465f6b704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2d97caee586418a8981108465f6b704 already in progress: creating tablet
W20251024 19:47:56.427791 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 8b2f85cccb274d9da74bb37f91f020af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b2f85cccb274d9da74bb37f91f020af already in progress: creating tablet
I20251024 19:47:56.428332 1775 tablet_replica.cc:333] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.428435 1776 ts_tablet_manager.cc:1929] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.428623 1776 log.cc:1199] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/9af5d5bd80a14f2ebdc7a595b17416e6
I20251024 19:47:56.428861 1775 raft_consensus.cc:2243] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.429052 1776 ts_tablet_manager.cc:1950] T 9af5d5bd80a14f2ebdc7a595b17416e6 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.429330 1775 raft_consensus.cc:2272] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.430240 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 9af5d5bd80a14f2ebdc7a595b17416e6 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]) successfully deleted
I20251024 19:47:56.430400 1775 ts_tablet_manager.cc:1916] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.430684 1776 tablet_replica.cc:333] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:56.430897 1776 raft_consensus.cc:2243] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.431159 1776 raft_consensus.cc:2272] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.431514 1768 tablet_bootstrap.cc:492] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.431708 1768 ts_tablet_manager.cc:1403] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.011s
I20251024 19:47:56.432071 1776 ts_tablet_manager.cc:1916] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.432065 1011 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskc9NInd/build/asan/bin/kudu
/tmp/dist-test-taskc9NInd/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.252.254:36505
--webserver_interface=127.0.252.254
--webserver_port=42051
--builtin_ntp_servers=127.0.252.212:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251024 19:47:56.432574 1768 raft_consensus.cc:359] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.432825 1768 raft_consensus.cc:385] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.432919 1768 raft_consensus.cc:740] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.433283 1768 consensus_queue.cc:260] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.438941 1768 ts_tablet_manager.cc:1434] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251024 19:47:56.439531 1768 tablet_bootstrap.cc:492] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.441637 1768 tablet_bootstrap.cc:654] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.444147 1776 ts_tablet_manager.cc:1929] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.444361 1776 log.cc:1199] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/7e31eef151884385854096ff71484a9c
I20251024 19:47:56.444829 1776 ts_tablet_manager.cc:1950] T 7e31eef151884385854096ff71484a9c P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.445994 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 7e31eef151884385854096ff71484a9c (table default.Table_dabc8d67782e4372b14bcf1862b25d3a [id=4e6695ccb7d342a6985b2a10a6bdfaac]) successfully deleted
I20251024 19:47:56.446693 1767 tablet_bootstrap.cc:492] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:56.446856 1767 ts_tablet_manager.cc:1403] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.002s
I20251024 19:47:56.447587 1767 raft_consensus.cc:359] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.447744 1767 raft_consensus.cc:385] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.447886 1767 raft_consensus.cc:740] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.448129 1767 consensus_queue.cc:260] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.449152 1767 ts_tablet_manager.cc:1434] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:47:56.449486 1767 tablet_bootstrap.cc:492] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:56.451493 1767 tablet_bootstrap.cc:654] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.452541 1775 ts_tablet_manager.cc:1929] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.453047 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a35e66c1df02471480ce2e87424f6e91" } new_table_name: "default.Table_f683ab67c4774e25861c7821970d1981" modify_external_catalogs: true
I20251024 19:47:56.455479 1775 log.cc:1199] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/44d777147a60462e86b3abf18a40f2c9
I20251024 19:47:56.455507 1768 tablet_bootstrap.cc:492] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.455642 1768 ts_tablet_manager.cc:1403] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20251024 19:47:56.455868 1775 ts_tablet_manager.cc:1950] T 44d777147a60462e86b3abf18a40f2c9 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.456307 1768 raft_consensus.cc:359] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.456544 1768 raft_consensus.cc:385] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.456709 1768 raft_consensus.cc:740] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.456980 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 44d777147a60462e86b3abf18a40f2c9 (table default.Table_7ddfa79850884c62900d2e92bc798c64 [id=750d9803961d4003b052aa142e097a45]) successfully deleted
I20251024 19:47:56.457020 1768 consensus_queue.cc:260] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.458684 1775 tablet_replica.cc:333] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.459203 1775 raft_consensus.cc:2243] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.459311 1768 ts_tablet_manager.cc:1434] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251024 19:47:56.459688 1775 raft_consensus.cc:2272] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.460489 1768 tablet_bootstrap.cc:492] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.461745 1775 ts_tablet_manager.cc:1916] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.462388 1768 tablet_bootstrap.cc:654] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.463261 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet 74c8c9949988403890a96ca535d367aa from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.463858 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6b2b5091b93348958149ec3640ed6971" } new_table_name: "default.Table_656be68a26b545d2b254abc2edb15254" modify_external_catalogs: true
I20251024 19:47:56.464156 1767 tablet_bootstrap.cc:492] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:56.464314 1767 ts_tablet_manager.cc:1403] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20251024 19:47:56.464963 1767 raft_consensus.cc:359] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.465117 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 44d777147a60462e86b3abf18a40f2c9 with delete_type TABLET_DATA_DELETED (replaced by tablet c70e51a4bd7c423aae7d6e2ddf0c60b4) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.465224 1767 raft_consensus.cc:385] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.465297 1767 raft_consensus.cc:740] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.465445 1776 tablet_replica.cc:333] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:56.465672 1776 raft_consensus.cc:2243] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.465693 1767 consensus_queue.cc:260] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.465883 1776 raft_consensus.cc:2272] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.467231 1767 ts_tablet_manager.cc:1434] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:47:56.467567 1776 ts_tablet_manager.cc:1916] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.468070 1767 tablet_bootstrap.cc:492] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:56.469791 1767 tablet_bootstrap.cc:654] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.470547 1775 ts_tablet_manager.cc:1929] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.470680 1775 log.cc:1199] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/20f32e34ce51446d8cc7d4938017e60e
I20251024 19:47:56.471020 1775 ts_tablet_manager.cc:1950] T 20f32e34ce51446d8cc7d4938017e60e P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.472014 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 20f32e34ce51446d8cc7d4938017e60e (table default.Table_835ab1cc0a82456f8282bce320dca4f8 [id=91d53300d972437d9efba6385dbef72c]) successfully deleted
I20251024 19:47:56.472216 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e31eef151884385854096ff71484a9c with delete_type TABLET_DATA_DELETED (replaced by tablet 7cbbedcba0f04ea6b00670e921d25200) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.473217 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf7d13f588543b1973364fb7e801ffa with delete_type TABLET_DATA_DELETED (replaced by tablet 1745bbc68d6846fab1a6197f898754b4) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.474685 1768 tablet_bootstrap.cc:492] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.474902 1768 ts_tablet_manager.cc:1403] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20251024 19:47:56.475767 1768 raft_consensus.cc:359] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.476047 1768 raft_consensus.cc:385] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.479588 1776 ts_tablet_manager.cc:1929] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.479786 1776 log.cc:1199] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/44d777147a60462e86b3abf18a40f2c9
I20251024 19:47:56.480195 1776 ts_tablet_manager.cc:1950] T 44d777147a60462e86b3abf18a40f2c9 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.481783 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 44d777147a60462e86b3abf18a40f2c9 (table default.Table_7ddfa79850884c62900d2e92bc798c64 [id=750d9803961d4003b052aa142e097a45]) successfully deleted
I20251024 19:47:56.476192 1768 raft_consensus.cc:740] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
W20251024 19:47:56.482434 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7e31eef151884385854096ff71484a9c because the tablet was not found. No further retry: Not found: Tablet not found: 7e31eef151884385854096ff71484a9c
I20251024 19:47:56.482625 1768 consensus_queue.cc:260] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.484156 1768 ts_tablet_manager.cc:1434] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251024 19:47:56.484560 1768 tablet_bootstrap.cc:492] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.485561 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_96c033f8e7ab4c919b38c80fabe3e367"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.486321 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96c033f8e7ab4c919b38c80fabe3e367 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.486900 1768 tablet_bootstrap.cc:654] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.488044 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet fab45ee691fe495196cf671e69368d4d from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.488339 1767 tablet_bootstrap.cc:492] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:56.488535 1767 ts_tablet_manager.cc:1403] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251024 19:47:56.488616 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet e2d97caee586418a8981108465f6b704 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.489205 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c4ffe38e352744a3891e5b1435036e9f" } new_table_name: "default.Table_69e288a9d6d24a5f841f62cd9a208f0f" modify_external_catalogs: true
I20251024 19:47:56.489524 1767 raft_consensus.cc:359] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.489758 1767 raft_consensus.cc:385] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.489871 1767 raft_consensus.cc:740] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.490146 1767 consensus_queue.cc:260] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.490648 1767 ts_tablet_manager.cc:1434] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251024 19:47:56.492395 1767 tablet_bootstrap.cc:492] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:56.494762 1767 tablet_bootstrap.cc:654] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.495888 1768 tablet_bootstrap.cc:492] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.496145 1768 ts_tablet_manager.cc:1403] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20251024 19:47:56.497318 1768 raft_consensus.cc:359] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:56.497653 1768 raft_consensus.cc:385] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.497848 1768 raft_consensus.cc:740] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.498275 1768 consensus_queue.cc:260] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:56.499680 1768 ts_tablet_manager.cc:1434] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:47:56.501730 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet fab45ee691fe495196cf671e69368d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.503103 1768 tablet_bootstrap.cc:492] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.505194 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 26d6f1105b324afba36081c760930ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.505722 1776 tablet_replica.cc:333] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:56.505918 1776 raft_consensus.cc:2243] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.506119 1776 raft_consensus.cc:2272] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.506608 1768 tablet_bootstrap.cc:654] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.506875 1776 ts_tablet_manager.cc:1916] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.509298 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf7d13f588543b1973364fb7e801ffa with delete_type TABLET_DATA_DELETED (replaced by tablet 1745bbc68d6846fab1a6197f898754b4) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.511690 1192 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet f1ad65fd080e452e8094975d420336c7 (table default.Table_c4539f71581f488cad3bc3b4b2f3a0eb [id=2fd5c541b06a46469e65ed0942d2c394]). Expected version 3 got 0
I20251024 19:47:56.512523 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_be4f56e3a9054dff8204c38760c34f8b" } modify_external_catalogs: true
I20251024 19:47:56.512712 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_be4f56e3a9054dff8204c38760c34f8b" } modify_external_catalogs: true
I20251024 19:47:56.513345 1776 ts_tablet_manager.cc:1929] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.513470 1776 log.cc:1199] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/26d6f1105b324afba36081c760930ef0
I20251024 19:47:56.514138 1776 ts_tablet_manager.cc:1950] T 26d6f1105b324afba36081c760930ef0 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.515216 1776 tablet_replica.cc:333] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:56.515496 1776 raft_consensus.cc:2243] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.515728 1776 raft_consensus.cc:2272] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.515899 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 26d6f1105b324afba36081c760930ef0 (table default.Table_2759324fce09406ab532d22228541f6e [id=30be39f295db49d98ee0c102dd6b3b14]) successfully deleted
I20251024 19:47:56.516430 1772 raft_consensus.cc:493] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:56.516630 1776 ts_tablet_manager.cc:1916] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.516618 1772 raft_consensus.cc:515] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:56.517431 1772 leader_election.cc:290] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:47:56.518163 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1ad65fd080e452e8094975d420336c7" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:56.518674 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1ad65fd080e452e8094975d420336c7" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:47:56.519210 1305 leader_election.cc:343] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:56.519253 1768 tablet_bootstrap.cc:492] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
W20251024 19:47:56.519793 1308 leader_election.cc:343] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:56.519888 1768 ts_tablet_manager.cc:1403] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20251024 19:47:56.519979 1308 leader_election.cc:304] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:56.520507 1772 raft_consensus.cc:2749] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:56.521296 1768 raft_consensus.cc:359] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:56.521663 1768 raft_consensus.cc:385] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.521821 1768 raft_consensus.cc:740] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.522150 1768 consensus_queue.cc:260] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:56.522943 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2d941eafb1414f068dc656946082f617
I20251024 19:47:56.523411 1191 catalog_manager.cc:6719] ReplaceTablet: tablet 2d941eafb1414f068dc656946082f617 deleted and replaced by tablet f91316c030dc4f7da27977a9dde01f4b
I20251024 19:47:56.523572 1768 ts_tablet_manager.cc:1434] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:47:56.523890 1768 tablet_bootstrap.cc:492] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.525101 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d941eafb1414f068dc656946082f617 with delete_type TABLET_DATA_DELETED (replaced by tablet f91316c030dc4f7da27977a9dde01f4b) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.525871 1776 ts_tablet_manager.cc:1929] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.525903 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d941eafb1414f068dc656946082f617 with delete_type TABLET_DATA_DELETED (replaced by tablet f91316c030dc4f7da27977a9dde01f4b) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.526116 1768 tablet_bootstrap.cc:654] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:47:56.526711 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2d941eafb1414f068dc656946082f617 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d941eafb1414f068dc656946082f617 already in progress: creating tablet
I20251024 19:47:56.529176 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d941eafb1414f068dc656946082f617 with delete_type TABLET_DATA_DELETED (replaced by tablet f91316c030dc4f7da27977a9dde01f4b) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.531126 1776 log.cc:1199] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/3cf7d13f588543b1973364fb7e801ffa
I20251024 19:47:56.531749 1776 ts_tablet_manager.cc:1950] T 3cf7d13f588543b1973364fb7e801ffa P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.533192 1191 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet f1ad65fd080e452e8094975d420336c7 (table default.Table_c4539f71581f488cad3bc3b4b2f3a0eb [id=2fd5c541b06a46469e65ed0942d2c394]). Expected version 3 got 0
I20251024 19:47:56.534127 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_85e6c4c107c94e7abfc7c3396aaf6e81" } modify_external_catalogs: true
I20251024 19:47:56.534350 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_85e6c4c107c94e7abfc7c3396aaf6e81" } modify_external_catalogs: true
I20251024 19:47:56.535190 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 3cf7d13f588543b1973364fb7e801ffa (table default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]) successfully deleted
W20251024 19:47:56.535508 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2d941eafb1414f068dc656946082f617 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d941eafb1414f068dc656946082f617 already in progress: creating tablet
I20251024 19:47:56.537176 1768 tablet_bootstrap.cc:492] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.537504 1768 ts_tablet_manager.cc:1403] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20251024 19:47:56.538398 1768 raft_consensus.cc:359] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.538617 1768 raft_consensus.cc:385] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.538766 1768 raft_consensus.cc:740] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.539032 1768 consensus_queue.cc:260] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.540364 1768 ts_tablet_manager.cc:1434] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251024 19:47:56.540688 1768 tablet_bootstrap.cc:492] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.542507 1768 tablet_bootstrap.cc:654] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.542783 1766 tablet_bootstrap.cc:492] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.543044 1766 ts_tablet_manager.cc:1403] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.151s user 0.036s sys 0.038s
I20251024 19:47:56.544044 1766 raft_consensus.cc:359] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.544371 1766 raft_consensus.cc:385] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.544553 1766 raft_consensus.cc:740] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.544947 1766 consensus_queue.cc:260] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.546693 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b96722ee06cf4e26a55a913c060a1209
I20251024 19:47:56.547262 1184 catalog_manager.cc:6719] ReplaceTablet: tablet b96722ee06cf4e26a55a913c060a1209 deleted and replaced by tablet 2e956ea914c545d79be6d632bfc4ec57
I20251024 19:47:56.549750 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet b96722ee06cf4e26a55a913c060a1209 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e956ea914c545d79be6d632bfc4ec57) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.550103 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet b96722ee06cf4e26a55a913c060a1209 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e956ea914c545d79be6d632bfc4ec57) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.551103 1768 tablet_bootstrap.cc:492] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
W20251024 19:47:56.551128 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b96722ee06cf4e26a55a913c060a1209 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b96722ee06cf4e26a55a913c060a1209 already in progress: creating tablet
I20251024 19:47:56.551321 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet b96722ee06cf4e26a55a913c060a1209 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e956ea914c545d79be6d632bfc4ec57) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.551889 1766 ts_tablet_manager.cc:1434] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251024 19:47:56.552635 1768 ts_tablet_manager.cc:1403] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
W20251024 19:47:56.553457 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b96722ee06cf4e26a55a913c060a1209 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b96722ee06cf4e26a55a913c060a1209 already in progress: creating tablet
I20251024 19:47:56.553416 1768 raft_consensus.cc:359] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.553651 1768 raft_consensus.cc:385] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.553745 1768 raft_consensus.cc:740] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.553977 1768 consensus_queue.cc:260] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.554610 1768 ts_tablet_manager.cc:1434] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251024 19:47:56.554771 1766 tablet_bootstrap.cc:492] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.555008 1768 tablet_bootstrap.cc:492] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.556592 1766 tablet_bootstrap.cc:654] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.557813 1768 tablet_bootstrap.cc:654] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.478358 1774 ts_tablet_manager.cc:1929] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.562119 1766 tablet_bootstrap.cc:492] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.562373 1766 ts_tablet_manager.cc:1403] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20251024 19:47:56.562916 1774 log.cc:1199] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/7e31eef151884385854096ff71484a9c
I20251024 19:47:56.563216 1766 raft_consensus.cc:359] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.563428 1766 raft_consensus.cc:385] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.563514 1766 raft_consensus.cc:740] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.563769 1766 consensus_queue.cc:260] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.565048 1766 ts_tablet_manager.cc:1434] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251024 19:47:56.563236 1768 tablet_bootstrap.cc:492] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.565810 1768 ts_tablet_manager.cc:1403] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20251024 19:47:56.566017 1774 ts_tablet_manager.cc:1950] T 7e31eef151884385854096ff71484a9c P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:56.567034 1766 tablet_bootstrap.cc:492] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.568859 1766 tablet_bootstrap.cc:654] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.571641 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a2fc8ac42bbc44af8bf194f2d3511bd5" } modify_external_catalogs: true
I20251024 19:47:56.571867 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a2fc8ac42bbc44af8bf194f2d3511bd5" } modify_external_catalogs: true
I20251024 19:47:56.572371 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 26d6f1105b324afba36081c760930ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.572944 1775 tablet_replica.cc:333] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.573130 1775 raft_consensus.cc:2243] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.573292 1775 raft_consensus.cc:2272] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.573899 1775 ts_tablet_manager.cc:1916] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.578420 1766 tablet_bootstrap.cc:492] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.578611 1766 ts_tablet_manager.cc:1403] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20251024 19:47:56.579923 1766 raft_consensus.cc:359] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.579123 1775 ts_tablet_manager.cc:1929] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.580221 1775 log.cc:1199] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/26d6f1105b324afba36081c760930ef0
I20251024 19:47:56.580667 1775 ts_tablet_manager.cc:1950] T 26d6f1105b324afba36081c760930ef0 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.581696 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 26d6f1105b324afba36081c760930ef0 (table default.Table_2759324fce09406ab532d22228541f6e [id=30be39f295db49d98ee0c102dd6b3b14]) successfully deleted
I20251024 19:47:56.566708 1768 raft_consensus.cc:359] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.582002 1768 raft_consensus.cc:385] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.582074 1766 raft_consensus.cc:385] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.582159 1768 raft_consensus.cc:740] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.582302 1766 raft_consensus.cc:740] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.582499 1768 consensus_queue.cc:260] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.582599 1766 consensus_queue.cc:260] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.583252 1768 ts_tablet_manager.cc:1434] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251024 19:47:56.583662 1768 tablet_bootstrap.cc:492] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.585781 1766 ts_tablet_manager.cc:1434] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20251024 19:47:56.588387 1774 tablet_replica.cc:333] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
W20251024 19:47:56.588454 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet fc8c01a4c07a42a0bc48a58313090e68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc8c01a4c07a42a0bc48a58313090e68 already in progress: creating tablet
I20251024 19:47:56.588568 1774 raft_consensus.cc:2243] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:47:56.588712 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet e2d97caee586418a8981108465f6b704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2d97caee586418a8981108465f6b704 already in progress: creating tablet
I20251024 19:47:56.589221 1768 tablet_bootstrap.cc:654] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:47:56.589354 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 8b2f85cccb274d9da74bb37f91f020af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b2f85cccb274d9da74bb37f91f020af already in progress: creating tablet
I20251024 19:47:56.590044 1766 tablet_bootstrap.cc:492] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.592692 1766 tablet_bootstrap.cc:654] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:47:56.597012 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 922f975a672746b3b72acf4d3050f688 consensus state has no leader
I20251024 19:47:56.598150 1766 tablet_bootstrap.cc:492] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.598367 1766 ts_tablet_manager.cc:1403] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20251024 19:47:56.599596 1766 raft_consensus.cc:359] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.599925 1766 raft_consensus.cc:385] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.600054 1766 raft_consensus.cc:740] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.600342 1766 consensus_queue.cc:260] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.601673 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_812ae4fbc0fc4fdea1317e0d9794c50b" } modify_external_catalogs: true
I20251024 19:47:56.601874 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_812ae4fbc0fc4fdea1317e0d9794c50b" } modify_external_catalogs: true
I20251024 19:47:56.604089 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 26d6f1105b324afba36081c760930ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.604938 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 9af5d5bd80a14f2ebdc7a595b17416e6 with delete_type TABLET_DATA_DELETED (replaced by tablet fb08e56db59b4db796c50db5bda93af9) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.605329 1775 tablet_replica.cc:333] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.605511 1775 raft_consensus.cc:2243] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.605690 1775 raft_consensus.cc:2272] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:47:56.606774 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 26d6f1105b324afba36081c760930ef0 because the tablet was not found. No further retry: Not found: Tablet not found: 26d6f1105b324afba36081c760930ef0
I20251024 19:47:56.607611 1775 ts_tablet_manager.cc:1916] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.610095 1772 raft_consensus.cc:493] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:56.610235 1772 raft_consensus.cc:515] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.610935 1772 leader_election.cc:290] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:56.611673 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf7d13f588543b1973364fb7e801ffa" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:47:56.612144 1305 leader_election.cc:343] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 3cf7d13f588543b1973364fb7e801ffa
W20251024 19:47:56.612377 1308 leader_election.cc:343] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:56.612535 1308 leader_election.cc:304] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:56.613129 1772 raft_consensus.cc:2749] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:56.588758 1774 raft_consensus.cc:2272] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.614445 1774 ts_tablet_manager.cc:1916] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.618786 1775 ts_tablet_manager.cc:1929] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.619446 1775 log.cc:1199] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/9af5d5bd80a14f2ebdc7a595b17416e6
I20251024 19:47:56.619951 1775 ts_tablet_manager.cc:1950] T 9af5d5bd80a14f2ebdc7a595b17416e6 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.621302 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 9af5d5bd80a14f2ebdc7a595b17416e6 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]) successfully deleted
I20251024 19:47:56.619297 1766 ts_tablet_manager.cc:1434] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20251024 19:47:56.624245 1774 ts_tablet_manager.cc:1929] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.624650 1766 tablet_bootstrap.cc:492] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.626427 1766 tablet_bootstrap.cc:654] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.632530 1774 log.cc:1199] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/74c8c9949988403890a96ca535d367aa
I20251024 19:47:56.633486 1768 tablet_bootstrap.cc:492] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.633773 1768 ts_tablet_manager.cc:1403] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.019s
I20251024 19:47:56.634799 1768 raft_consensus.cc:359] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.635051 1768 raft_consensus.cc:385] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.635140 1768 raft_consensus.cc:740] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.636317 1768 consensus_queue.cc:260] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.637354 1768 ts_tablet_manager.cc:1434] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251024 19:47:56.638059 1768 tablet_bootstrap.cc:492] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.638486 1772 raft_consensus.cc:493] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:56.638677 1772 raft_consensus.cc:515] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:56.639542 1772 leader_election.cc:290] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:47:56.640214 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07d66fe64bff4a00a15554f212456698" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:56.640393 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07d66fe64bff4a00a15554f212456698" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:47:56.640764 1308 leader_election.cc:343] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:47:56.641180 1305 leader_election.cc:343] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:56.641381 1305 leader_election.cc:304] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:56.642037 1772 raft_consensus.cc:2749] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:47:56.642760 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb17a35d8d8446cf9870d05e107ed237 consensus state has no leader
I20251024 19:47:56.643239 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cf69470650f643c195eb5fbecbd2d2c4" } modify_external_catalogs: true
I20251024 19:47:56.643436 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cf69470650f643c195eb5fbecbd2d2c4" } modify_external_catalogs: true
W20251024 19:47:56.644158 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c consensus state has no leader
I20251024 19:47:56.644567 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3d57d480ef4c4a1492438c1b7dd6135c" } new_table_name: "default.Table_0680f9139a8a436a80fce09c095912ce" modify_external_catalogs: true
I20251024 19:47:56.644609 1768 tablet_bootstrap.cc:654] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.646488 1774 ts_tablet_manager.cc:1950] T 74c8c9949988403890a96ca535d367aa P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:56.647970 1766 tablet_bootstrap.cc:492] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.649328 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 74c8c9949988403890a96ca535d367aa (table default.Table_36406152efad4ca4b1a106ed81d39fbe [id=6d9d362a97bd4c6ba8ccc960082ed831]) successfully deleted
I20251024 19:47:56.649430 1766 ts_tablet_manager.cc:1403] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.016s
I20251024 19:47:56.650522 1766 raft_consensus.cc:359] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.650782 1766 raft_consensus.cc:385] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.650902 1766 raft_consensus.cc:740] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.651162 1766 consensus_queue.cc:260] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.651686 1766 ts_tablet_manager.cc:1434] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:47:56.651942 1774 tablet_replica.cc:333] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:56.652091 1766 tablet_bootstrap.cc:492] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.652246 1774 raft_consensus.cc:2243] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.653327 1774 raft_consensus.cc:2272] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.653170 1768 tablet_bootstrap.cc:492] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.653689 1768 ts_tablet_manager.cc:1403] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20251024 19:47:56.653920 1766 tablet_bootstrap.cc:654] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.654647 1767 tablet_bootstrap.cc:492] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:56.654588 1768 raft_consensus.cc:359] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.654776 1768 raft_consensus.cc:385] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.654831 1767 ts_tablet_manager.cc:1403] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.162s user 0.048s sys 0.081s
I20251024 19:47:56.654874 1768 raft_consensus.cc:740] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.655176 1768 consensus_queue.cc:260] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.655642 1767 raft_consensus.cc:359] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.655844 1767 raft_consensus.cc:385] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.655932 1767 raft_consensus.cc:740] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.655937 1768 ts_tablet_manager.cc:1434] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:47:56.656389 1768 tablet_bootstrap.cc:492] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.656226 1767 consensus_queue.cc:260] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.656973 1767 ts_tablet_manager.cc:1434] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:47:56.657550 1767 tablet_bootstrap.cc:492] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:56.659672 1767 tablet_bootstrap.cc:654] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.662264 1774 ts_tablet_manager.cc:1916] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.662407 1766 tablet_bootstrap.cc:492] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.662930 1766 ts_tablet_manager.cc:1403] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
W20251024 19:47:56.663249 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f396c2982f8d487cb44c9422b09da14a consensus state has no leader
I20251024 19:47:56.663638 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f02338f14f2b44b3b89757b7edc59e32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.663833 1766 raft_consensus.cc:359] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.664005 1766 raft_consensus.cc:385] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.664084 1766 raft_consensus.cc:740] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
W20251024 19:47:56.664228 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f02338f14f2b44b3b89757b7edc59e32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.664315 1766 consensus_queue.cc:260] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.665690 1766 ts_tablet_manager.cc:1434] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251024 19:47:56.666023 1766 tablet_bootstrap.cc:492] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.668980 1766 tablet_bootstrap.cc:654] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.671069 1767 tablet_bootstrap.cc:492] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:56.671236 1767 ts_tablet_manager.cc:1403] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20251024 19:47:56.672091 1767 raft_consensus.cc:359] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.672317 1767 raft_consensus.cc:385] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.672412 1767 raft_consensus.cc:740] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.672638 1767 consensus_queue.cc:260] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.673126 1767 ts_tablet_manager.cc:1434] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251024 19:47:56.673434 1767 tablet_bootstrap.cc:492] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:56.675277 1767 tablet_bootstrap.cc:654] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.679749 1192 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 211015ec811a4ce681372e516443ad24
I20251024 19:47:56.682318 1184 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 07d66fe64bff4a00a15554f212456698 (table default.Table_7c34d3a8affd4e69a31b276ccbe096ae [id=aadca5a1b6d4494aa37f8a8cc76ad12a]). Expected version 3 got 0
I20251024 19:47:56.685624 1766 tablet_bootstrap.cc:492] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
W20251024 19:47:56.685791 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9820e68ad5264df1a79b22f3b5369c5a has no consensus state
I20251024 19:47:56.685866 1766 ts_tablet_manager.cc:1403] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20251024 19:47:56.686056 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet 0ffaa4997edd4b1187579958037ea793 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.686702 1766 raft_consensus.cc:359] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:56.687005 1766 raft_consensus.cc:385] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.687170 1766 raft_consensus.cc:740] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.687398 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 74c8c9949988403890a96ca535d367aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.687529 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet b96722ee06cf4e26a55a913c060a1209 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e956ea914c545d79be6d632bfc4ec57) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.687800 1775 tablet_replica.cc:333] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.687744 1766 consensus_queue.cc:260] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:56.688052 1775 raft_consensus.cc:2243] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.688242 1775 raft_consensus.cc:2272] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.688623 1766 ts_tablet_manager.cc:1434] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:47:56.688853 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 3caecdff61ce48f7b7c3846bb380d21e (DEFAULT_TABLE table=default.Table_9a372068f5b34bdcbe40f464da2797a1 [id=e3c15f9790304d4eb7be44889ef89d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.688989 1766 tablet_bootstrap.cc:492] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.689388 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3caecdff61ce48f7b7c3846bb380d21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.689442 1775 ts_tablet_manager.cc:1916] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.689522 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 3caecdff61ce48f7b7c3846bb380d21e (DEFAULT_TABLE table=default.Table_9a372068f5b34bdcbe40f464da2797a1 [id=e3c15f9790304d4eb7be44889ef89d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.689968 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3caecdff61ce48f7b7c3846bb380d21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.698705 1775 ts_tablet_manager.cc:1929] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.698654 1187 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet f396c2982f8d487cb44c9422b09da14a (table default.Table_69e288a9d6d24a5f841f62cd9a208f0f [id=969d428d45db42d18a5a9b82ec945bfb]). Expected version 1 got 0
I20251024 19:47:56.701807 1775 log.cc:1199] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/74c8c9949988403890a96ca535d367aa
I20251024 19:47:56.702301 1775 ts_tablet_manager.cc:1950] T 74c8c9949988403890a96ca535d367aa P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.702741 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 3caecdff61ce48f7b7c3846bb380d21e (DEFAULT_TABLE table=default.Table_9a372068f5b34bdcbe40f464da2797a1 [id=e3c15f9790304d4eb7be44889ef89d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.703159 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3caecdff61ce48f7b7c3846bb380d21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.703774 1775 tablet_replica.cc:333] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.705281 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 74c8c9949988403890a96ca535d367aa (table default.Table_36406152efad4ca4b1a106ed81d39fbe [id=6d9d362a97bd4c6ba8ccc960082ed831]) successfully deleted
I20251024 19:47:56.705498 1775 raft_consensus.cc:2243] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.705693 1775 raft_consensus.cc:2272] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.706465 1775 ts_tablet_manager.cc:1916] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.712421 1768 tablet_bootstrap.cc:654] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.705278 1766 tablet_bootstrap.cc:654] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.715540 1775 ts_tablet_manager.cc:1929] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.716099 1775 log.cc:1199] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/b96722ee06cf4e26a55a913c060a1209
I20251024 19:47:56.716639 1775 ts_tablet_manager.cc:1950] T b96722ee06cf4e26a55a913c060a1209 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.718274 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet b96722ee06cf4e26a55a913c060a1209 (table default.Table_4dda32c7ff144630bebd572cf5f9f822 [id=350b37bf42194fcfbb660bffe83e47c4]) successfully deleted
I20251024 19:47:56.720930 1774 ts_tablet_manager.cc:1929] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.723491 1774 log.cc:1199] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/3cf7d13f588543b1973364fb7e801ffa
I20251024 19:47:56.724236 1774 ts_tablet_manager.cc:1950] T 3cf7d13f588543b1973364fb7e801ffa P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:56.728790 1767 tablet_bootstrap.cc:492] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:56.729045 1767 ts_tablet_manager.cc:1403] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.011s
I20251024 19:47:56.730058 1767 raft_consensus.cc:359] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.730365 1767 raft_consensus.cc:385] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.730517 1767 raft_consensus.cc:740] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.730870 1767 consensus_queue.cc:260] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.731002 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b35aa67b3a714699afde1d56713e2644"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.731765 1767 ts_tablet_manager.cc:1434] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251024 19:47:56.731876 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b35aa67b3a714699afde1d56713e2644 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.732087 1767 tablet_bootstrap.cc:492] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:56.732601 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0bc77b2f9654428484956c26da42bf54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.733107 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bc77b2f9654428484956c26da42bf54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.733232 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_eb9c8019451d46fba31a87a0b6ac6ad8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.733661 1767 tablet_bootstrap.cc:654] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.735517 1766 tablet_bootstrap.cc:492] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.735740 1766 ts_tablet_manager.cc:1403] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.013s
I20251024 19:47:56.736667 1766 raft_consensus.cc:359] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:56.736948 1766 raft_consensus.cc:385] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.737079 1766 raft_consensus.cc:740] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.737423 1766 consensus_queue.cc:260] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:47:56.738231 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb9c8019451d46fba31a87a0b6ac6ad8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.738998 1766 ts_tablet_manager.cc:1434] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:47:56.739523 1766 tablet_bootstrap.cc:492] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.745179 1766 tablet_bootstrap.cc:654] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.749218 1768 tablet_bootstrap.cc:492] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.749542 1768 ts_tablet_manager.cc:1403] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.004s
I20251024 19:47:56.749742 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 74c8c9949988403890a96ca535d367aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.750089 1776 tablet_replica.cc:333] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:56.750274 1776 raft_consensus.cc:2243] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.750511 1776 raft_consensus.cc:2272] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.750694 1779 raft_consensus.cc:493] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:56.750900 1779 raft_consensus.cc:515] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.751312 1776 ts_tablet_manager.cc:1916] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.751731 1779 leader_election.cc:290] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:47:56.755939 1768 raft_consensus.cc:359] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.756166 1768 raft_consensus.cc:385] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.756253 1768 raft_consensus.cc:740] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.756538 1768 consensus_queue.cc:260] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.756659 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed6c66a4ce141fbabbee58877689439" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:56.757068 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed6c66a4ce141fbabbee58877689439" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:47:56.757258 1768 ts_tablet_manager.cc:1434] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251024 19:47:56.757599 1768 tablet_bootstrap.cc:492] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:47:56.757840 1444 leader_election.cc:343] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:56.758397 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 3cf7d13f588543b1973364fb7e801ffa (table default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]) successfully deleted
I20251024 19:47:56.757277 1371 raft_consensus.cc:2468] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:47:56.759112 1447 leader_election.cc:304] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:47:56.759640 1779 raft_consensus.cc:2804] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:47:56.759750 1779 raft_consensus.cc:493] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:47:56.759848 1779 raft_consensus.cc:3060] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:56.760955 1774 tablet_replica.cc:333] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:56.761129 1774 raft_consensus.cc:2243] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.761314 1774 raft_consensus.cc:2272] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.762208 1774 ts_tablet_manager.cc:1916] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.762785 1779 raft_consensus.cc:515] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.763355 1779 leader_election.cc:290] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:47:56.764128 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed6c66a4ce141fbabbee58877689439" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:47:56.764969 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ed6c66a4ce141fbabbee58877689439" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:47:56.765148 1371 raft_consensus.cc:3060] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:56.767542 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ed6c66a4ce141fbabbee58877689439 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.767843 1775 tablet_replica.cc:333] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.768010 1775 raft_consensus.cc:2243] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.768164 1768 tablet_bootstrap.cc:654] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.768182 1775 raft_consensus.cc:2272] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.769018 1775 ts_tablet_manager.cc:1916] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:47:56.769801 1444 leader_election.cc:343] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:56.770231 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 97c6dce647064344b0b53a7ffaeafb75
I20251024 19:47:56.770816 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 047436af5f6d4f688bd83e33089d3ee4
I20251024 19:47:56.771711 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 97c6dce647064344b0b53a7ffaeafb75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.772451 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 97c6dce647064344b0b53a7ffaeafb75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.772446 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 047436af5f6d4f688bd83e33089d3ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.772773 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 97c6dce647064344b0b53a7ffaeafb75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.773619 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 047436af5f6d4f688bd83e33089d3ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.773769 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 047436af5f6d4f688bd83e33089d3ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.773993 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 889bd73511b945b4a4e751f60929a5dc
I20251024 19:47:56.774727 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_99584fc98c7e4f0f853914cc3dda2b12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.775328 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99584fc98c7e4f0f853914cc3dda2b12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.775404 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c3cbf88afafa464099c69d8e017743e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.775875 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3cbf88afafa464099c69d8e017743e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.776475 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 0ffaa4997edd4b1187579958037ea793
I20251024 19:47:56.777117 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet fab45ee691fe495196cf671e69368d4d from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.777627 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet 1745bbc68d6846fab1a6197f898754b4 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.778404 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet 07d66fe64bff4a00a15554f212456698 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.781502 1767 tablet_bootstrap.cc:492] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:56.781669 1767 ts_tablet_manager.cc:1403] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.050s user 0.018s sys 0.023s
I20251024 19:47:56.782083 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 889bd73511b945b4a4e751f60929a5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.782984 1767 raft_consensus.cc:359] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.783170 1767 raft_consensus.cc:385] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.783278 1767 raft_consensus.cc:740] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.783504 1767 consensus_queue.cc:260] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.783789 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet f396c2982f8d487cb44c9422b09da14a from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:56.784607 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 20f32e34ce51446d8cc7d4938017e60e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.785136 1767 ts_tablet_manager.cc:1434] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:47:56.785705 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ffaa4997edd4b1187579958037ea793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.786324 1767 tablet_bootstrap.cc:492] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:56.788890 1767 tablet_bootstrap.cc:654] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.789156 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a17ef21a2da140b4954cf16abc6d2e95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.789722 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 889bd73511b945b4a4e751f60929a5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.789954 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ffaa4997edd4b1187579958037ea793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.790218 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet b96722ee06cf4e26a55a913c060a1209 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e956ea914c545d79be6d632bfc4ec57) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.776997 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e629fad818da4d4aa6cf7b4d30895738"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.790786 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e629fad818da4d4aa6cf7b4d30895738 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.790784 1774 ts_tablet_manager.cc:1929] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.790896 1774 log.cc:1199] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/fab45ee691fe495196cf671e69368d4d
I20251024 19:47:56.791282 1774 ts_tablet_manager.cc:1950] T fab45ee691fe495196cf671e69368d4d P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:56.792387 1768 tablet_bootstrap.cc:492] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.792431 1774 tablet_replica.cc:333] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:56.792541 1768 ts_tablet_manager.cc:1403] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.003s
I20251024 19:47:56.792665 1774 raft_consensus.cc:2243] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.792855 1774 raft_consensus.cc:2272] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.793406 1774 ts_tablet_manager.cc:1916] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.793376 1768 raft_consensus.cc:359] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.793591 1768 raft_consensus.cc:385] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.793687 1768 raft_consensus.cc:740] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.793993 1768 consensus_queue.cc:260] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.794663 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 889bd73511b945b4a4e751f60929a5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.794960 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf7d13f588543b1973364fb7e801ffa with delete_type TABLET_DATA_DELETED (replaced by tablet 1745bbc68d6846fab1a6197f898754b4) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.795504 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet b96722ee06cf4e26a55a913c060a1209 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e956ea914c545d79be6d632bfc4ec57) from {username='slave'} at 127.0.0.1:46462
W20251024 19:47:56.795759 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a17ef21a2da140b4954cf16abc6d2e95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.795897 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ffaa4997edd4b1187579958037ea793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.797588 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet fab45ee691fe495196cf671e69368d4d (table default.Table_46d6218260e8421b9093ea1630d5c8de [id=79ccc02d30c743d0a448971d552b35dd]) successfully deleted
I20251024 19:47:56.797734 1768 ts_tablet_manager.cc:1434] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
W20251024 19:47:56.797837 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2d941eafb1414f068dc656946082f617 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d941eafb1414f068dc656946082f617 already in progress: creating tablet
I20251024 19:47:56.798694 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ed6c66a4ce141fbabbee58877689439 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.798884 1768 tablet_bootstrap.cc:492] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.798928 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b2f85cccb274d9da74bb37f91f020af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.800701 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9a372068f5b34bdcbe40f464da2797a1" } new_table_name: "default.Table_5cb242202328456ca4c831b963804e06" modify_external_catalogs: true
I20251024 19:47:56.802286 1768 tablet_bootstrap.cc:654] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.806268 1775 ts_tablet_manager.cc:1929] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.807217 1371 raft_consensus.cc:2468] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:47:56.810892 1767 tablet_bootstrap.cc:492] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:56.811098 1767 ts_tablet_manager.cc:1403] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.008s
W20251024 19:47:56.811522 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3caecdff61ce48f7b7c3846bb380d21e consensus state has no leader
I20251024 19:47:56.812074 1767 raft_consensus.cc:359] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.812283 1767 raft_consensus.cc:385] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.812376 1767 raft_consensus.cc:740] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.812745 1776 ts_tablet_manager.cc:1929] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.812925 1776 log.cc:1199] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/74c8c9949988403890a96ca535d367aa
I20251024 19:47:56.813428 1776 ts_tablet_manager.cc:1950] T 74c8c9949988403890a96ca535d367aa P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.813647 1775 log.cc:1199] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/4ed6c66a4ce141fbabbee58877689439
I20251024 19:47:56.814132 1775 ts_tablet_manager.cc:1950] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.814682 1768 tablet_bootstrap.cc:492] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.814920 1768 ts_tablet_manager.cc:1403] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20251024 19:47:56.815250 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 74c8c9949988403890a96ca535d367aa (table default.Table_36406152efad4ca4b1a106ed81d39fbe [id=6d9d362a97bd4c6ba8ccc960082ed831]) successfully deleted
W20251024 19:47:56.815526 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 97c6dce647064344b0b53a7ffaeafb75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97c6dce647064344b0b53a7ffaeafb75 already in progress: creating tablet
I20251024 19:47:56.815691 1776 tablet_replica.cc:333] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:47:56.815789 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 047436af5f6d4f688bd83e33089d3ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 047436af5f6d4f688bd83e33089d3ee4 already in progress: creating tablet
I20251024 19:47:56.815910 1776 raft_consensus.cc:2243] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:47:56.816018 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 889bd73511b945b4a4e751f60929a5dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 889bd73511b945b4a4e751f60929a5dc already in progress: creating tablet
I20251024 19:47:56.816075 1776 raft_consensus.cc:2272] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.810236 1447 leader_election.cc:304] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
W20251024 19:47:56.816622 1447 raft_consensus.cc:2686] T 4ed6c66a4ce141fbabbee58877689439 P 10a9536207e343f188e0ef709081d04e: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251024 19:47:56.817032 1768 raft_consensus.cc:359] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.817224 1768 raft_consensus.cc:385] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.817298 1768 raft_consensus.cc:740] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.817337 1775 tablet_replica.cc:333] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.817636 1775 raft_consensus.cc:2243] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.817566 1768 consensus_queue.cc:260] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.817835 1775 raft_consensus.cc:2272] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.807672 1774 ts_tablet_manager.cc:1929] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.818095 1774 log.cc:1199] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/b96722ee06cf4e26a55a913c060a1209
I20251024 19:47:56.818230 1768 ts_tablet_manager.cc:1434] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:47:56.818224 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 4ed6c66a4ce141fbabbee58877689439 (table default.Table_c5b5dcab83194aa28ae6465fbaa7f023 [id=c112bf97e1fa4c21ac91f769bdfc6f2f]) successfully deleted
W20251024 19:47:56.818475 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 047436af5f6d4f688bd83e33089d3ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 047436af5f6d4f688bd83e33089d3ee4 already in progress: creating tablet
I20251024 19:47:56.818547 1774 ts_tablet_manager.cc:1950] T b96722ee06cf4e26a55a913c060a1209 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:47:56.818708 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 97c6dce647064344b0b53a7ffaeafb75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97c6dce647064344b0b53a7ffaeafb75 already in progress: creating tablet
I20251024 19:47:56.818790 1768 tablet_bootstrap.cc:492] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:47:56.818929 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 889bd73511b945b4a4e751f60929a5dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 889bd73511b945b4a4e751f60929a5dc already in progress: creating tablet
I20251024 19:47:56.819371 1775 ts_tablet_manager.cc:1916] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.819551 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_446bc99dda0d455c9362702ac0c2c685"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:56.820075 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_446bc99dda0d455c9362702ac0c2c685 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.820952 1766 tablet_bootstrap.cc:492] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:56.821099 1766 ts_tablet_manager.cc:1403] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.003s
I20251024 19:47:56.812682 1767 consensus_queue.cc:260] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.825910 1767 ts_tablet_manager.cc:1434] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.015s user 0.001s sys 0.002s
I20251024 19:47:56.826305 1767 tablet_bootstrap.cc:492] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:56.828053 1767 tablet_bootstrap.cc:654] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.828801 1768 tablet_bootstrap.cc:654] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.830060 1766 raft_consensus.cc:359] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.830628 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet fab45ee691fe495196cf671e69368d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.830962 1776 ts_tablet_manager.cc:1916] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.830435 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet b96722ee06cf4e26a55a913c060a1209 (table default.Table_4dda32c7ff144630bebd572cf5f9f822 [id=350b37bf42194fcfbb660bffe83e47c4]) successfully deleted
I20251024 19:47:56.830277 1766 raft_consensus.cc:385] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.830694 1775 ts_tablet_manager.cc:1929] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.831990 1775 log.cc:1199] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/3cf7d13f588543b1973364fb7e801ffa
I20251024 19:47:56.831948 1766 raft_consensus.cc:740] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
W20251024 19:47:56.831806 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 97c6dce647064344b0b53a7ffaeafb75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97c6dce647064344b0b53a7ffaeafb75 already in progress: creating tablet
I20251024 19:47:56.832438 1775 ts_tablet_manager.cc:1950] T 3cf7d13f588543b1973364fb7e801ffa P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.832350 1766 consensus_queue.cc:260] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:47:56.833993 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 047436af5f6d4f688bd83e33089d3ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 047436af5f6d4f688bd83e33089d3ee4 already in progress: creating tablet
I20251024 19:47:56.834503 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 3cf7d13f588543b1973364fb7e801ffa (table default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]) successfully deleted
I20251024 19:47:56.835216 1774 tablet_replica.cc:333] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:56.835420 1774 raft_consensus.cc:2243] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.835610 1774 raft_consensus.cc:2272] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.836118 1192 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 922f975a672746b3b72acf4d3050f688 (table default.Table_6b632546d5924db7afe499fc1b1b9db0 [id=eab377d5ce1343d7a12c3740fe46beb5]). Expected version 2 got 0
I20251024 19:47:56.836580 1774 ts_tablet_manager.cc:1916] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.837621 1766 ts_tablet_manager.cc:1434] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251024 19:47:56.840134 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 15c08dcf379f4df9a0e461d6dcfeeac4 (DEFAULT_TABLE table=default.Table_96c033f8e7ab4c919b38c80fabe3e367 [id=473fc6762db94b959bb9a260b11140f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.840540 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15c08dcf379f4df9a0e461d6dcfeeac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.840842 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 15c08dcf379f4df9a0e461d6dcfeeac4 (DEFAULT_TABLE table=default.Table_96c033f8e7ab4c919b38c80fabe3e367 [id=473fc6762db94b959bb9a260b11140f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.845172 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 15c08dcf379f4df9a0e461d6dcfeeac4 (DEFAULT_TABLE table=default.Table_96c033f8e7ab4c919b38c80fabe3e367 [id=473fc6762db94b959bb9a260b11140f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.845463 1776 ts_tablet_manager.cc:1929] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.845644 1776 log.cc:1199] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/20f32e34ce51446d8cc7d4938017e60e
I20251024 19:47:56.845686 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15c08dcf379f4df9a0e461d6dcfeeac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.846058 1776 ts_tablet_manager.cc:1950] T 20f32e34ce51446d8cc7d4938017e60e P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.846585 1348 tablet_service.cc:1505] Processing CreateTablet for tablet cc0bf7aee1a1421b9d985020a1674639 (DEFAULT_TABLE table=default.Table_272323b2c4ac4c8cb0ff412a632ee43d [id=48bed8c74a444f9fa83d52387a2650c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.846845 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 8721ee81e4e24153a0aad8ce99f872e1 (DEFAULT_TABLE table=default.Table_9291fbfe8e484be094f28ad94814b13f [id=7ec9391c52b1424eb52850f2c6ba4779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.847019 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc0bf7aee1a1421b9d985020a1674639. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.847270 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8721ee81e4e24153a0aad8ce99f872e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.849921 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 95d7fbfa8ae54da49dd5603294faa7d3 (DEFAULT_TABLE table=default.Table_c772dc06afa04976b56055df70060bb3 [id=fb0e7cab20f44aaebdb09862bd193f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.850342 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d7fbfa8ae54da49dd5603294faa7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.845662 1774 ts_tablet_manager.cc:1929] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.853091 1774 log.cc:1199] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/889bd73511b945b4a4e751f60929a5dc
I20251024 19:47:56.853643 1774 ts_tablet_manager.cc:1950] T 889bd73511b945b4a4e751f60929a5dc P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:56.853942 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 110907ae943f45b5bcaa9b66ccdb84fd (DEFAULT_TABLE table=default.Table_2cbb0aad326e4bae85d4c5cb833150c6 [id=1acdc5d0bfb94e0faf47055b20d803b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.854375 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 110907ae943f45b5bcaa9b66ccdb84fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.854738 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 8721ee81e4e24153a0aad8ce99f872e1 (DEFAULT_TABLE table=default.Table_9291fbfe8e484be094f28ad94814b13f [id=7ec9391c52b1424eb52850f2c6ba4779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.854735 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 8721ee81e4e24153a0aad8ce99f872e1 (DEFAULT_TABLE table=default.Table_9291fbfe8e484be094f28ad94814b13f [id=7ec9391c52b1424eb52850f2c6ba4779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.855105 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8721ee81e4e24153a0aad8ce99f872e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.855376 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 95d7fbfa8ae54da49dd5603294faa7d3 (DEFAULT_TABLE table=default.Table_c772dc06afa04976b56055df70060bb3 [id=fb0e7cab20f44aaebdb09862bd193f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.855724 1779 raft_consensus.cc:493] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:56.855857 1779 raft_consensus.cc:515] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.856021 1776 tablet_replica.cc:333] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:56.856216 1776 raft_consensus.cc:2243] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.856415 1776 raft_consensus.cc:2272] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.856952 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f396c2982f8d487cb44c9422b09da14a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:56.857074 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 95d7fbfa8ae54da49dd5603294faa7d3 (DEFAULT_TABLE table=default.Table_c772dc06afa04976b56055df70060bb3 [id=fb0e7cab20f44aaebdb09862bd193f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.857206 1650 raft_consensus.cc:2468] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:47:56.857450 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d7fbfa8ae54da49dd5603294faa7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.857748 1776 ts_tablet_manager.cc:1916] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.857864 1444 leader_election.cc:304] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:47:56.858176 1819 raft_consensus.cc:2804] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:47:56.858294 1819 raft_consensus.cc:493] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:47:56.858340 1779 leader_election.cc:290] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:47:56.858423 1819 raft_consensus.cc:3060] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:56.858959 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f396c2982f8d487cb44c9422b09da14a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:47:56.859179 1371 raft_consensus.cc:2468] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:47:56.860980 1484 tablet_service.cc:1505] Processing CreateTablet for tablet cc0bf7aee1a1421b9d985020a1674639 (DEFAULT_TABLE table=default.Table_272323b2c4ac4c8cb0ff412a632ee43d [id=48bed8c74a444f9fa83d52387a2650c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.861325 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b35aa67b3a714699afde1d56713e2644" } new_table_name: "default.Table_5760bec3ecb94c9c94b3fd2a7ecdd4fd" modify_external_catalogs: true
I20251024 19:47:56.862208 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 20f32e34ce51446d8cc7d4938017e60e (table default.Table_835ab1cc0a82456f8282bce320dca4f8 [id=91d53300d972437d9efba6385dbef72c]) successfully deleted
W20251024 19:47:56.862533 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 0ffaa4997edd4b1187579958037ea793 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ffaa4997edd4b1187579958037ea793 already in progress: creating tablet
I20251024 19:47:56.862550 1819 raft_consensus.cc:515] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.863021 1623 tablet_service.cc:1505] Processing CreateTablet for tablet cc0bf7aee1a1421b9d985020a1674639 (DEFAULT_TABLE table=default.Table_272323b2c4ac4c8cb0ff412a632ee43d [id=48bed8c74a444f9fa83d52387a2650c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.863463 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc0bf7aee1a1421b9d985020a1674639. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.863680 1819 leader_election.cc:290] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:47:56.863931 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f396c2982f8d487cb44c9422b09da14a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:47:56.864688 1650 raft_consensus.cc:3060] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:56.865025 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f396c2982f8d487cb44c9422b09da14a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:47:56.865263 1371 raft_consensus.cc:3060] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:56.868857 1371 raft_consensus.cc:2468] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:47:56.869665 1447 leader_election.cc:304] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:47:56.870039 1819 raft_consensus.cc:2804] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:47:56.870223 1819 raft_consensus.cc:697] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:47:56.870664 1819 consensus_queue.cc:237] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.871201 1774 tablet_replica.cc:333] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:56.871599 1774 raft_consensus.cc:2243] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.871902 1774 raft_consensus.cc:2272] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.874375 1187 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet f1ad65fd080e452e8094975d420336c7 (table default.Table_c4539f71581f488cad3bc3b4b2f3a0eb [id=2fd5c541b06a46469e65ed0942d2c394]). Expected version 3 got 0
I20251024 19:47:56.876305 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 07d66fe64bff4a00a15554f212456698
I20251024 19:47:56.877219 1774 ts_tablet_manager.cc:1916] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.877324 1186 catalog_manager.cc:6719] ReplaceTablet: tablet 07d66fe64bff4a00a15554f212456698 deleted and replaced by tablet 2346c03802a245a2af8c086b626d660d
I20251024 19:47:56.879411 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5cb242202328456ca4c831b963804e06" } new_table_name: "default.Table_8c169b038d764be684a4e5b56faf745b" modify_external_catalogs: true
I20251024 19:47:56.881323 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b7db01944072476f9c48985f48b2be65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:56.881477 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 07d66fe64bff4a00a15554f212456698 with delete_type TABLET_DATA_DELETED (replaced by tablet 2346c03802a245a2af8c086b626d660d) from {username='slave'} at 127.0.0.1:54246
W20251024 19:47:56.881922 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7db01944072476f9c48985f48b2be65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:56.883549 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet e2d97caee586418a8981108465f6b704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.883865 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d941eafb1414f068dc656946082f617 with delete_type TABLET_DATA_DELETED (replaced by tablet f91316c030dc4f7da27977a9dde01f4b) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.884299 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 07d66fe64bff4a00a15554f212456698 with delete_type TABLET_DATA_DELETED (replaced by tablet 2346c03802a245a2af8c086b626d660d) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.885358 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 889bd73511b945b4a4e751f60929a5dc (table default.Table_a2fc8ac42bbc44af8bf194f2d3511bd5 [id=5518afdf09da4433bee8f2221ead84c1]) successfully deleted
I20251024 19:47:56.886579 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 07d66fe64bff4a00a15554f212456698 with delete_type TABLET_DATA_DELETED (replaced by tablet 2346c03802a245a2af8c086b626d660d) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.887148 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 07d66fe64bff4a00a15554f212456698 with delete_type TABLET_DATA_DELETED (replaced by tablet 2346c03802a245a2af8c086b626d660d) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.888033 1650 raft_consensus.cc:2468] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:47:56.889079 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6b632546d5924db7afe499fc1b1b9db0" } new_table_name: "default.Table_ab42d318a6cc4626be3d62dda5948bf0" modify_external_catalogs: true
I20251024 19:47:56.891160 1774 ts_tablet_manager.cc:1929] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.895181 1774 log.cc:1199] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/0ffaa4997edd4b1187579958037ea793
I20251024 19:47:56.896025 1774 ts_tablet_manager.cc:1950] T 0ffaa4997edd4b1187579958037ea793 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:56.896013 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 110907ae943f45b5bcaa9b66ccdb84fd (DEFAULT_TABLE table=default.Table_2cbb0aad326e4bae85d4c5cb833150c6 [id=1acdc5d0bfb94e0faf47055b20d803b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.896620 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 1745bbc68d6846fab1a6197f898754b4
I20251024 19:47:56.897042 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_01b0d0ffefb744f3a1abe0eeab079dc2" } new_table_name: "default.Table_ea60f0e4363c406d9ccff5f16b8ecce9" modify_external_catalogs: true
I20251024 19:47:56.897220 1190 catalog_manager.cc:6719] ReplaceTablet: tablet 1745bbc68d6846fab1a6197f898754b4 deleted and replaced by tablet 31647a48fb834c80b6920c756a94250d
I20251024 19:47:56.898124 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_59f512ddad6842eca4865d6387dee631" } modify_external_catalogs: true
I20251024 19:47:56.898464 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_59f512ddad6842eca4865d6387dee631" } modify_external_catalogs: true
I20251024 19:47:56.899055 1774 tablet_replica.cc:333] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:56.899588 1774 raft_consensus.cc:2243] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.899818 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 1745bbc68d6846fab1a6197f898754b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 31647a48fb834c80b6920c756a94250d) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.900346 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 1745bbc68d6846fab1a6197f898754b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 31647a48fb834c80b6920c756a94250d) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.900679 1774 raft_consensus.cc:2272] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.900925 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 0ffaa4997edd4b1187579958037ea793 (table default.Table_cf69470650f643c195eb5fbecbd2d2c4 [id=c1c8743e95d34655b1a8779920ed837a]) successfully deleted
I20251024 19:47:56.902066 1774 ts_tablet_manager.cc:1916] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.903477 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15c08dcf379f4df9a0e461d6dcfeeac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.903606 1766 tablet_bootstrap.cc:492] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:56.903831 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc0bf7aee1a1421b9d985020a1674639. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.904254 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8721ee81e4e24153a0aad8ce99f872e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.905500 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d7fbfa8ae54da49dd5603294faa7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.906311 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 110907ae943f45b5bcaa9b66ccdb84fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.898070 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 1745bbc68d6846fab1a6197f898754b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 31647a48fb834c80b6920c756a94250d) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.917738 1766 tablet_bootstrap.cc:654] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.917848 1774 ts_tablet_manager.cc:1929] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.918018 1774 log.cc:1199] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/4ed6c66a4ce141fbabbee58877689439
I20251024 19:47:56.920006 1774 ts_tablet_manager.cc:1950] T 4ed6c66a4ce141fbabbee58877689439 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:56.921949 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f396c2982f8d487cb44c9422b09da14a
I20251024 19:47:56.922497 1184 catalog_manager.cc:6719] ReplaceTablet: tablet f396c2982f8d487cb44c9422b09da14a deleted and replaced by tablet e20881e400cf4fac914b70afc79c084f
I20251024 19:47:56.922722 1774 tablet_replica.cc:333] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
W20251024 19:47:56.923169 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c04082bd1b444b468c177bef239701c6 has no consensus state
I20251024 19:47:56.923293 1774 raft_consensus.cc:2243] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.923491 1774 raft_consensus.cc:2272] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.923560 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet f396c2982f8d487cb44c9422b09da14a with delete_type TABLET_DATA_DELETED (replaced by tablet e20881e400cf4fac914b70afc79c084f) from {username='slave'} at 127.0.0.1:54246
W20251024 19:47:56.923717 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3caecdff61ce48f7b7c3846bb380d21e consensus state has no leader
W20251024 19:47:56.924144 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 922f975a672746b3b72acf4d3050f688 consensus state has no leader
I20251024 19:47:56.924695 1768 tablet_bootstrap.cc:492] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.924868 1768 ts_tablet_manager.cc:1403] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.106s user 0.017s sys 0.011s
I20251024 19:47:56.925940 1768 raft_consensus.cc:359] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.926167 1768 raft_consensus.cc:385] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.926276 1768 raft_consensus.cc:740] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.926653 1768 consensus_queue.cc:260] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:56.927340 1776 ts_tablet_manager.cc:1929] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.928040 1768 ts_tablet_manager.cc:1434] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:47:56.928088 1782 raft_consensus.cc:493] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:56.928324 1768 tablet_bootstrap.cc:492] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.928251 1782 raft_consensus.cc:515] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.928895 1782 leader_election.cc:290] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:47:56.929647 1776 log.cc:1199] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/b96722ee06cf4e26a55a913c060a1209
I20251024 19:47:56.929730 1772 raft_consensus.cc:493] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:56.929865 1772 raft_consensus.cc:515] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.929958 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fab45ee691fe495196cf671e69368d4d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:56.930099 1776 ts_tablet_manager.cc:1950] T b96722ee06cf4e26a55a913c060a1209 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.930231 1511 raft_consensus.cc:2468] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:47:56.930454 1772 leader_election.cc:290] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:56.931062 1776 tablet_replica.cc:333] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:56.931229 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2d97caee586418a8981108465f6b704" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:47:56.931782 1308 leader_election.cc:343] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:56.931975 1768 tablet_bootstrap.cc:654] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.932363 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet b96722ee06cf4e26a55a913c060a1209 (table default.Table_4dda32c7ff144630bebd572cf5f9f822 [id=350b37bf42194fcfbb660bffe83e47c4]) successfully deleted
I20251024 19:47:56.931298 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2d97caee586418a8981108465f6b704" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:56.934540 1776 raft_consensus.cc:2243] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.934703 1776 raft_consensus.cc:2272] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:47:56.934813 1305 leader_election.cc:343] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:56.934953 1305 leader_election.cc:304] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:56.935374 1772 raft_consensus.cc:2749] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:56.935506 1776 ts_tablet_manager.cc:1916] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.932113 1586 leader_election.cc:304] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:47:56.935802 1586 raft_consensus.cc:2686] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251024 19:47:56.936040 1586 leader_election.cc:343] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: fab45ee691fe495196cf671e69368d4d
I20251024 19:47:56.938400 1186 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet fb17a35d8d8446cf9870d05e107ed237 (table default.Table_f683ab67c4774e25861c7821970d1981 [id=9a05bcf7ad5d430c9ef6d6e18a27808e]). Expected version 1 got 0
W20251024 19:47:56.940925 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae86e74f810c4b989ac633f59e6ac445 consensus state has no leader
I20251024 19:47:56.942585 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 0397637f0778442a871528d5dd173689
I20251024 19:47:56.945647 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 0397637f0778442a871528d5dd173689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:56.946630 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet fb17a35d8d8446cf9870d05e107ed237 (table default.Table_f683ab67c4774e25861c7821970d1981 [id=9a05bcf7ad5d430c9ef6d6e18a27808e]). Expected version 1 got 0
I20251024 19:47:56.949808 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet f396c2982f8d487cb44c9422b09da14a with delete_type TABLET_DATA_DELETED (replaced by tablet e20881e400cf4fac914b70afc79c084f) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.951296 1776 ts_tablet_manager.cc:1929] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.951537 1776 log.cc:1199] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/fab45ee691fe495196cf671e69368d4d
I20251024 19:47:56.952013 1776 ts_tablet_manager.cc:1950] T fab45ee691fe495196cf671e69368d4d P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.952172 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 0397637f0778442a871528d5dd173689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.952697 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet f396c2982f8d487cb44c9422b09da14a with delete_type TABLET_DATA_DELETED (replaced by tablet e20881e400cf4fac914b70afc79c084f) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:56.953192 1775 tablet_replica.cc:333] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.953397 1775 raft_consensus.cc:2243] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.953660 1775 raft_consensus.cc:2272] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.954337 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 0397637f0778442a871528d5dd173689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:56.955288 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet fab45ee691fe495196cf671e69368d4d (table default.Table_46d6218260e8421b9093ea1630d5c8de [id=79ccc02d30c743d0a448971d552b35dd]) successfully deleted
W20251024 19:47:56.955633 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 07d66fe64bff4a00a15554f212456698 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07d66fe64bff4a00a15554f212456698 already in progress: creating tablet
I20251024 19:47:56.955722 1768 tablet_bootstrap.cc:492] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.955879 1768 ts_tablet_manager.cc:1403] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.010s
W20251024 19:47:56.955884 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 1745bbc68d6846fab1a6197f898754b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1745bbc68d6846fab1a6197f898754b4 already in progress: creating tablet
I20251024 19:47:56.956761 1768 raft_consensus.cc:359] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.957003 1768 raft_consensus.cc:385] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.957096 1768 raft_consensus.cc:740] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.957374 1768 consensus_queue.cc:260] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.957495 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 2e956ea914c545d79be6d632bfc4ec57 (DEFAULT_TABLE table=default.Table_4dda32c7ff144630bebd572cf5f9f822 [id=350b37bf42194fcfbb660bffe83e47c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.957940 1768 ts_tablet_manager.cc:1434] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:47:56.957993 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e956ea914c545d79be6d632bfc4ec57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.958333 1768 tablet_bootstrap.cc:492] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:56.961809 1776 tablet_replica.cc:333] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:56.962289 1776 raft_consensus.cc:2243] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:56.962517 1776 raft_consensus.cc:2272] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:56.963521 1776 ts_tablet_manager.cc:1916] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.964110 1186 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c (table default.Table_656be68a26b545d2b254abc2edb15254 [id=a1c4569aa6934d4a99787cd9e3240edb]). Expected version 1 got 0
I20251024 19:47:56.966403 1775 ts_tablet_manager.cc:1916] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:56.971309 1768 tablet_bootstrap.cc:654] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:47:56.971326 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b96722ee06cf4e26a55a913c060a1209 because the tablet was not found. No further retry: Not found: Tablet not found: b96722ee06cf4e26a55a913c060a1209
W20251024 19:47:56.971637 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 0ffaa4997edd4b1187579958037ea793 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ffaa4997edd4b1187579958037ea793 already in progress: creating tablet
I20251024 19:47:56.972007 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 4ed6c66a4ce141fbabbee58877689439 (table default.Table_c5b5dcab83194aa28ae6465fbaa7f023 [id=c112bf97e1fa4c21ac91f769bdfc6f2f]) successfully deleted
I20251024 19:47:56.977979 1767 tablet_bootstrap.cc:492] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:56.978420 1767 ts_tablet_manager.cc:1403] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.152s user 0.030s sys 0.012s
I20251024 19:47:56.978919 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 9820e68ad5264df1a79b22f3b5369c5a (DEFAULT_TABLE table=default.Table_0680f9139a8a436a80fce09c095912ce [id=071147c9baed46f8a30f4c2fbc0fbfaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:56.979403 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9820e68ad5264df1a79b22f3b5369c5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:56.980232 1767 raft_consensus.cc:359] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:56.981637 1767 raft_consensus.cc:385] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.982096 1767 raft_consensus.cc:740] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.982841 1767 consensus_queue.cc:260] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:56.983961 1767 ts_tablet_manager.cc:1434] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251024 19:47:56.984838 1767 tablet_bootstrap.cc:492] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:56.987040 1767 tablet_bootstrap.cc:654] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:56.991109 1776 ts_tablet_manager.cc:1929] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.991243 1776 log.cc:1199] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/f396c2982f8d487cb44c9422b09da14a
I20251024 19:47:56.991593 1776 ts_tablet_manager.cc:1950] T f396c2982f8d487cb44c9422b09da14a P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:56.994040 1775 ts_tablet_manager.cc:1929] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:56.994210 1775 log.cc:1199] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/07d66fe64bff4a00a15554f212456698
I20251024 19:47:56.994777 1775 ts_tablet_manager.cc:1950] T 07d66fe64bff4a00a15554f212456698 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:56.995179 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet f396c2982f8d487cb44c9422b09da14a (table default.Table_69e288a9d6d24a5f841f62cd9a208f0f [id=969d428d45db42d18a5a9b82ec945bfb]) successfully deleted
W20251024 19:47:56.995689 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 0397637f0778442a871528d5dd173689 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0397637f0778442a871528d5dd173689 already in progress: creating tablet
I20251024 19:47:56.996308 1768 tablet_bootstrap.cc:492] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:56.996471 1768 ts_tablet_manager.cc:1403] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20251024 19:47:56.997272 1768 raft_consensus.cc:359] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.997470 1768 raft_consensus.cc:385] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:56.997866 1768 raft_consensus.cc:740] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:56.997997 1775 tablet_replica.cc:333] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:56.998180 1768 consensus_queue.cc:260] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:56.999332 1767 tablet_bootstrap.cc:492] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:56.999725 1775 raft_consensus.cc:2243] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:47:56.999907 1767 ts_tablet_manager.cc:1403] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20251024 19:47:57.000367 1768 ts_tablet_manager.cc:1434] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251024 19:47:57.000300 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 110907ae943f45b5bcaa9b66ccdb84fd (DEFAULT_TABLE table=default.Table_2cbb0aad326e4bae85d4c5cb833150c6 [id=1acdc5d0bfb94e0faf47055b20d803b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.000773 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 110907ae943f45b5bcaa9b66ccdb84fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.001447 1766 tablet_bootstrap.cc:492] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:57.001600 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 2e956ea914c545d79be6d632bfc4ec57 (DEFAULT_TABLE table=default.Table_4dda32c7ff144630bebd572cf5f9f822 [id=350b37bf42194fcfbb660bffe83e47c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.001691 1766 ts_tablet_manager.cc:1403] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.098s user 0.015s sys 0.014s
I20251024 19:47:57.002013 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e956ea914c545d79be6d632bfc4ec57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.002615 1766 raft_consensus.cc:359] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.002882 1766 raft_consensus.cc:385] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.003005 1766 raft_consensus.cc:740] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.003346 1766 consensus_queue.cc:260] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.004319 1775 pending_rounds.cc:70] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e: Trying to abort 1 pending ops.
I20251024 19:47:57.004608 1775 pending_rounds.cc:77] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429294069661696 op_type: NO_OP noop_request { }
I20251024 19:47:57.005108 1775 raft_consensus.cc:2889] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:47:57.005328 1775 raft_consensus.cc:2272] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.007194 1766 ts_tablet_manager.cc:1434] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251024 19:47:57.008934 1768 tablet_bootstrap.cc:492] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:57.010001 1766 tablet_bootstrap.cc:492] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:57.010756 1186 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet f1ad65fd080e452e8094975d420336c7 (table default.Table_c4539f71581f488cad3bc3b4b2f3a0eb [id=2fd5c541b06a46469e65ed0942d2c394]). Expected version 3 got 0
I20251024 19:47:57.010828 1768 tablet_bootstrap.cc:654] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.011989 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 2e956ea914c545d79be6d632bfc4ec57 (DEFAULT_TABLE table=default.Table_4dda32c7ff144630bebd572cf5f9f822 [id=350b37bf42194fcfbb660bffe83e47c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.012564 1766 tablet_bootstrap.cc:654] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.012861 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e956ea914c545d79be6d632bfc4ec57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.013674 1767 raft_consensus.cc:359] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:57.014497 1767 raft_consensus.cc:385] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.014664 1767 raft_consensus.cc:740] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.015233 1768 tablet_bootstrap.cc:492] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:57.015141 1767 consensus_queue.cc:260] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:57.015420 1768 ts_tablet_manager.cc:1403] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20251024 19:47:57.015774 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ea60f0e4363c406d9ccff5f16b8ecce9" } modify_external_catalogs: true
I20251024 19:47:57.015990 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ea60f0e4363c406d9ccff5f16b8ecce9" } modify_external_catalogs: true
I20251024 19:47:57.016201 1189 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet f1ad65fd080e452e8094975d420336c7 (table default.Table_c4539f71581f488cad3bc3b4b2f3a0eb [id=2fd5c541b06a46469e65ed0942d2c394]). Expected version 3 got 0
I20251024 19:47:57.016953 1767 ts_tablet_manager.cc:1434] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251024 19:47:57.017421 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_96c033f8e7ab4c919b38c80fabe3e367" } modify_external_catalogs: true
I20251024 19:47:57.017602 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_96c033f8e7ab4c919b38c80fabe3e367" } modify_external_catalogs: true
I20251024 19:47:57.017756 1767 tablet_bootstrap.cc:492] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:57.020550 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 979dd87a8c66480ea314fcc75fa878c3 (DEFAULT_TABLE table=default.Table_b728c3d6fcf44b4593b7883fcf4ea07d [id=cf70cd6b11614a0d8b39b707311c46fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.020843 1767 tablet_bootstrap.cc:654] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.020944 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 979dd87a8c66480ea314fcc75fa878c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.022822 1774 ts_tablet_manager.cc:1916] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.027231 1768 raft_consensus.cc:359] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.027449 1768 raft_consensus.cc:385] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.027542 1768 raft_consensus.cc:740] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.028707 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ae86e74f810c4b989ac633f59e6ac445
I20251024 19:47:57.028553 1768 consensus_queue.cc:260] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.029461 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 15c08dcf379f4df9a0e461d6dcfeeac4
I20251024 19:47:57.030192 1772 raft_consensus.cc:493] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:57.030287 1767 tablet_bootstrap.cc:492] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:57.030328 1772 raft_consensus.cc:515] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.030611 1767 ts_tablet_manager.cc:1403] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20251024 19:47:57.030973 1772 leader_election.cc:290] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:57.032035 1767 raft_consensus.cc:359] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:57.032275 1767 raft_consensus.cc:385] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.032460 1767 raft_consensus.cc:740] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.032912 1767 consensus_queue.cc:260] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:57.033872 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d941eafb1414f068dc656946082f617" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:57.034724 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d941eafb1414f068dc656946082f617" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:57.034864 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet ae86e74f810c4b989ac633f59e6ac445 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.035192 1616 tablet_service.cc:1505] Processing CreateTablet for tablet b2ea545b2cc14f589eda9419e55234bf (DEFAULT_TABLE table=default.Table_74349eed6d4b4e8e95ae01909bbb91e9 [id=94b8daec91b6478f9dc43734542246e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.035594 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2ea545b2cc14f589eda9419e55234bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:57.035698 1305 leader_election.cc:343] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.036935 1782 raft_consensus.cc:493] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:57.037089 1782 raft_consensus.cc:515] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:57.037871 1782 leader_election.cc:290] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:47:57.038012 1308 leader_election.cc:343] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.038151 1308 leader_election.cc:304] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:57.038771 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07d66fe64bff4a00a15554f212456698" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:47:57.038982 1371 raft_consensus.cc:2468] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
W20251024 19:47:57.039104 1586 leader_election.cc:343] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 07d66fe64bff4a00a15554f212456698
W20251024 19:47:57.039476 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet ae86e74f810c4b989ac633f59e6ac445 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae86e74f810c4b989ac633f59e6ac445 already in progress: creating tablet
I20251024 19:47:57.039618 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ed6c66a4ce141fbabbee58877689439 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.039949 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 07d66fe64bff4a00a15554f212456698 with delete_type TABLET_DATA_DELETED (replaced by tablet 2346c03802a245a2af8c086b626d660d) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.035262 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet 15c08dcf379f4df9a0e461d6dcfeeac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.040220 1776 tablet_replica.cc:333] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:57.040360 1772 raft_consensus.cc:2749] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:57.040467 1776 raft_consensus.cc:2243] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:57.040652 1776 raft_consensus.cc:2272] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.040906 1612 tablet_service.cc:1505] Processing CreateTablet for tablet f91316c030dc4f7da27977a9dde01f4b (DEFAULT_TABLE table=default.Table_ee4ef17d5f314379846d30a477d4911d [id=a402387542654ccf8e0b04ca8ecebb1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.042141 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 608696ec90ed45e1ad939ed2bfc88d34 (table default.Table_6e1fbc007b464e8c93803ba262a19fea [id=2c92cb09b287410c90b811e4b88c2260]). Expected version 1 got 0
W20251024 19:47:57.045817 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4ed6c66a4ce141fbabbee58877689439 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ed6c66a4ce141fbabbee58877689439 already in progress: creating tablet
I20251024 19:47:57.046590 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2e71745e53a04c5b8701ff72aedb203c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.047327 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e71745e53a04c5b8701ff72aedb203c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.048487 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c700d3a5bb0c4963ace8624ae2ef882d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:57.048597 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_673f2f271b3e4f9d9c5f6802a3646f8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:57.049222 1775 ts_tablet_manager.cc:1916] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.048487 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_41507c038ecb4e8ba8aadd563d12b6bf" } modify_external_catalogs: true
I20251024 19:47:57.048127 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_99584fc98c7e4f0f853914cc3dda2b12" } modify_external_catalogs: true
I20251024 19:47:57.050606 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f91316c030dc4f7da27977a9dde01f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.056763 1775 ts_tablet_manager.cc:1929] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:47:57.041217 1586 leader_election.cc:304] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
W20251024 19:47:57.057056 1586 raft_consensus.cc:2686] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251024 19:47:57.041476 1776 ts_tablet_manager.cc:1916] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.059728 1768 ts_tablet_manager.cc:1434] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.044s user 0.003s sys 0.003s
I20251024 19:47:57.060923 1768 tablet_bootstrap.cc:492] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:57.064975 1768 tablet_bootstrap.cc:654] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.066340 1776 ts_tablet_manager.cc:1929] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.066363 1774 ts_tablet_manager.cc:1929] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.066527 1774 log.cc:1199] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/8b2f85cccb274d9da74bb37f91f020af
I20251024 19:47:57.041731 1767 ts_tablet_manager.cc:1434] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20251024 19:47:57.067095 1767 tablet_bootstrap.cc:492] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:57.068316 1776 log.cc:1199] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/07d66fe64bff4a00a15554f212456698
I20251024 19:47:57.069134 1776 ts_tablet_manager.cc:1950] T 07d66fe64bff4a00a15554f212456698 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:57.065286 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet fab45ee691fe495196cf671e69368d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.069595 1775 log.cc:1199] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/f396c2982f8d487cb44c9422b09da14a
I20251024 19:47:57.070086 1774 ts_tablet_manager.cc:1950] T 8b2f85cccb274d9da74bb37f91f020af P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:57.070897 1775 ts_tablet_manager.cc:1950] T f396c2982f8d487cb44c9422b09da14a P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:57.072588 1774 tablet_replica.cc:333] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
W20251024 19:47:57.073200 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c700d3a5bb0c4963ace8624ae2ef882d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.073757 1774 raft_consensus.cc:2243] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:57.073998 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 07d66fe64bff4a00a15554f212456698 (table default.Table_7c34d3a8affd4e69a31b276ccbe096ae [id=aadca5a1b6d4494aa37f8a8cc76ad12a]) successfully deleted
W20251024 19:47:57.074285 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 15c08dcf379f4df9a0e461d6dcfeeac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15c08dcf379f4df9a0e461d6dcfeeac4 already in progress: creating tablet
I20251024 19:47:57.074471 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet 95d7fbfa8ae54da49dd5603294faa7d3 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:57.074561 1767 tablet_bootstrap.cc:654] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.074692 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_41507c038ecb4e8ba8aadd563d12b6bf" } modify_external_catalogs: true
W20251024 19:47:57.075179 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_673f2f271b3e4f9d9c5f6802a3646f8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.075325 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_99584fc98c7e4f0f853914cc3dda2b12" } modify_external_catalogs: true
I20251024 19:47:57.076283 1775 tablet_replica.cc:333] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:57.076560 1775 raft_consensus.cc:2243] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:57.076869 1775 raft_consensus.cc:2272] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.077162 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c564241bf74c49deaa26052b5c1d5afe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:57.077435 1774 raft_consensus.cc:2272] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.078174 1775 ts_tablet_manager.cc:1916] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.078377 1768 tablet_bootstrap.cc:492] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:57.078557 1774 ts_tablet_manager.cc:1916] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.078589 1768 ts_tablet_manager.cc:1403] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20251024 19:47:57.079619 1768 raft_consensus.cc:359] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.079836 1768 raft_consensus.cc:385] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.079972 1768 raft_consensus.cc:740] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.080260 1768 consensus_queue.cc:260] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.080953 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_66accd4024cc4cd98f10a25df857d785"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:57.083253 1768 ts_tablet_manager.cc:1434] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251024 19:47:57.084203 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ed6c66a4ce141fbabbee58877689439 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:55 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:47:57.081568 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66accd4024cc4cd98f10a25df857d785 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:57.085122 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c564241bf74c49deaa26052b5c1d5afe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.085312 1776 tablet_replica.cc:333] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:57.085315 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f96cb6877d844a50aacb91514a539fee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:57.085688 1776 raft_consensus.cc:2243] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:47:57.085891 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f96cb6877d844a50aacb91514a539fee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.085916 1776 raft_consensus.cc:2272] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.086899 1768 tablet_bootstrap.cc:492] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:57.087149 1776 ts_tablet_manager.cc:1916] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.087146 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 07d66fe64bff4a00a15554f212456698 (table default.Table_7c34d3a8affd4e69a31b276ccbe096ae [id=aadca5a1b6d4494aa37f8a8cc76ad12a]) successfully deleted
W20251024 19:47:57.087572 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 07d66fe64bff4a00a15554f212456698 because the tablet was not found. No further retry: Not found: Tablet not found: 07d66fe64bff4a00a15554f212456698
W20251024 19:47:57.087795 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 1745bbc68d6846fab1a6197f898754b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1745bbc68d6846fab1a6197f898754b4 already in progress: creating tablet
W20251024 19:47:57.088256 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 0397637f0778442a871528d5dd173689 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0397637f0778442a871528d5dd173689 already in progress: creating tablet
I20251024 19:47:57.088622 1768 tablet_bootstrap.cc:654] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.088621 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet f396c2982f8d487cb44c9422b09da14a (table default.Table_69e288a9d6d24a5f841f62cd9a208f0f [id=969d428d45db42d18a5a9b82ec945bfb]) successfully deleted
I20251024 19:47:57.089897 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet ae86e74f810c4b989ac633f59e6ac445 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.090437 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 9820e68ad5264df1a79b22f3b5369c5a (DEFAULT_TABLE table=default.Table_0680f9139a8a436a80fce09c095912ce [id=071147c9baed46f8a30f4c2fbc0fbfaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.090734 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 9820e68ad5264df1a79b22f3b5369c5a (DEFAULT_TABLE table=default.Table_0680f9139a8a436a80fce09c095912ce [id=071147c9baed46f8a30f4c2fbc0fbfaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.090852 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9820e68ad5264df1a79b22f3b5369c5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.091094 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9820e68ad5264df1a79b22f3b5369c5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.093959 1352 tablet_service.cc:1505] Processing CreateTablet for tablet f91316c030dc4f7da27977a9dde01f4b (DEFAULT_TABLE table=default.Table_ee4ef17d5f314379846d30a477d4911d [id=a402387542654ccf8e0b04ca8ecebb1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.094168 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 979dd87a8c66480ea314fcc75fa878c3 (DEFAULT_TABLE table=default.Table_b728c3d6fcf44b4593b7883fcf4ea07d [id=cf70cd6b11614a0d8b39b707311c46fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.094305 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f91316c030dc4f7da27977a9dde01f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.094508 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 979dd87a8c66480ea314fcc75fa878c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.095340 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 979dd87a8c66480ea314fcc75fa878c3 (DEFAULT_TABLE table=default.Table_b728c3d6fcf44b4593b7883fcf4ea07d [id=cf70cd6b11614a0d8b39b707311c46fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.095814 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 979dd87a8c66480ea314fcc75fa878c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.097857 1491 tablet_service.cc:1505] Processing CreateTablet for tablet b2ea545b2cc14f589eda9419e55234bf (DEFAULT_TABLE table=default.Table_74349eed6d4b4e8e95ae01909bbb91e9 [id=94b8daec91b6478f9dc43734542246e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.098225 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2ea545b2cc14f589eda9419e55234bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.098905 1342 tablet_service.cc:1505] Processing CreateTablet for tablet b2ea545b2cc14f589eda9419e55234bf (DEFAULT_TABLE table=default.Table_74349eed6d4b4e8e95ae01909bbb91e9 [id=94b8daec91b6478f9dc43734542246e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.099269 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2ea545b2cc14f589eda9419e55234bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.103305 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 15c08dcf379f4df9a0e461d6dcfeeac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.105468 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 8b2f85cccb274d9da74bb37f91f020af (table default.Table_8d9e77c4fcbf4a6e9f90d1c111a8cb50 [id=9b044de638ad4f14b886d199bc4a2532]) successfully deleted
I20251024 19:47:57.107163 1776 ts_tablet_manager.cc:1929] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.107304 1776 log.cc:1199] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/4ed6c66a4ce141fbabbee58877689439
I20251024 19:47:57.107638 1776 ts_tablet_manager.cc:1950] T 4ed6c66a4ce141fbabbee58877689439 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:57.109241 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 4ed6c66a4ce141fbabbee58877689439 (table default.Table_c5b5dcab83194aa28ae6465fbaa7f023 [id=c112bf97e1fa4c21ac91f769bdfc6f2f]) successfully deleted
I20251024 19:47:57.112303 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet fc8c01a4c07a42a0bc48a58313090e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.114491 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 15c08dcf379f4df9a0e461d6dcfeeac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.118139 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet ae86e74f810c4b989ac633f59e6ac445 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.122087 1767 tablet_bootstrap.cc:492] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:57.122967 1767 ts_tablet_manager.cc:1403] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.003s
I20251024 19:47:57.123802 1767 raft_consensus.cc:359] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.124051 1767 raft_consensus.cc:385] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.124135 1767 raft_consensus.cc:740] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.124182 1768 tablet_bootstrap.cc:492] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:57.124419 1767 consensus_queue.cc:260] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.126888 1767 ts_tablet_manager.cc:1434] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251024 19:47:57.127389 1767 tablet_bootstrap.cc:492] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:57.130513 1767 tablet_bootstrap.cc:654] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.131847 1483 tablet_service.cc:1505] Processing CreateTablet for tablet f91316c030dc4f7da27977a9dde01f4b (DEFAULT_TABLE table=default.Table_ee4ef17d5f314379846d30a477d4911d [id=a402387542654ccf8e0b04ca8ecebb1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.132326 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f91316c030dc4f7da27977a9dde01f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.132313 1768 ts_tablet_manager.cc:1403] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.004s
I20251024 19:47:57.133296 1768 raft_consensus.cc:359] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.133524 1768 raft_consensus.cc:385] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.133622 1768 raft_consensus.cc:740] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.134044 1768 consensus_queue.cc:260] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.140303 1766 tablet_bootstrap.cc:492] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:57.140486 1766 ts_tablet_manager.cc:1403] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.131s user 0.002s sys 0.007s
I20251024 19:47:57.141481 1766 raft_consensus.cc:359] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.141721 1766 raft_consensus.cc:385] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.141819 1766 raft_consensus.cc:740] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.142062 1766 consensus_queue.cc:260] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.142845 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_46b7ffb5bf6f45f2be4d1b242f43bfc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:57.143160 1775 ts_tablet_manager.cc:1929] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.143294 1775 log.cc:1199] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/fab45ee691fe495196cf671e69368d4d
W20251024 19:47:57.143536 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46b7ffb5bf6f45f2be4d1b242f43bfc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.143740 1775 ts_tablet_manager.cc:1950] T fab45ee691fe495196cf671e69368d4d P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:57.143764 1766 ts_tablet_manager.cc:1434] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251024 19:47:57.144048 1766 tablet_bootstrap.cc:492] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:57.145718 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet fab45ee691fe495196cf671e69368d4d (table default.Table_46d6218260e8421b9093ea1630d5c8de [id=79ccc02d30c743d0a448971d552b35dd]) successfully deleted
I20251024 19:47:57.146628 1775 tablet_replica.cc:333] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:57.146840 1775 raft_consensus.cc:2243] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:57.147030 1775 raft_consensus.cc:2272] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.147372 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 889bd73511b945b4a4e751f60929a5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.148195 1767 tablet_bootstrap.cc:492] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:57.148378 1767 ts_tablet_manager.cc:1403] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251024 19:47:57.149186 1767 raft_consensus.cc:359] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.149379 1767 raft_consensus.cc:385] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.149449 1767 raft_consensus.cc:740] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.149794 1767 consensus_queue.cc:260] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.155090 1767 ts_tablet_manager.cc:1434] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20251024 19:47:57.157248 1767 tablet_bootstrap.cc:492] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:57.148062 1766 tablet_bootstrap.cc:654] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.161257 1767 tablet_bootstrap.cc:654] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.163668 1775 ts_tablet_manager.cc:1916] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.167220 1819 raft_consensus.cc:493] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:57.167420 1865 raft_consensus.cc:493] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:57.167465 1819 raft_consensus.cc:515] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.168426 1819 leader_election.cc:290] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:57.169863 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "889bd73511b945b4a4e751f60929a5dc" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:47:57.170661 1444 leader_election.cc:343] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.172444 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8c5e539b72dd4150adca95d805acce44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.173110 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c5e539b72dd4150adca95d805acce44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.174667 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b728c3d6fcf44b4593b7883fcf4ea07d" } new_table_name: "default.Table_d5728217650a46e482b97caa10fea9a2" modify_external_catalogs: true
I20251024 19:47:57.176147 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2d74dcbcd9f249e5b10e1986cc0c0d18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.176710 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d74dcbcd9f249e5b10e1986cc0c0d18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.180866 1192 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 922f975a672746b3b72acf4d3050f688 (table default.Table_ab42d318a6cc4626be3d62dda5948bf0 [id=eab377d5ce1343d7a12c3740fe46beb5]). Expected version 3 got 0
I20251024 19:47:57.181496 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet 110907ae943f45b5bcaa9b66ccdb84fd from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:57.181735 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 07d66fe64bff4a00a15554f212456698 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:57.182338 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 2d941eafb1414f068dc656946082f617 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:57.183100 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b9caeb0e904f44dc8f879c8988a34678"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.183663 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9caeb0e904f44dc8f879c8988a34678 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.184280 1491 tablet_service.cc:1505] Processing CreateTablet for tablet c292ce055def4964ab143252c6b2b87e (DEFAULT_TABLE table=default.Table_446bc99dda0d455c9362702ac0c2c685 [id=10833938cf88466b834698420d7c01e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.184775 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c292ce055def4964ab143252c6b2b87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.184820 1183 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 608696ec90ed45e1ad939ed2bfc88d34 (table default.Table_6e1fbc007b464e8c93803ba262a19fea [id=2c92cb09b287410c90b811e4b88c2260]). Expected version 1 got 0
I20251024 19:47:57.185031 1483 tablet_service.cc:1505] Processing CreateTablet for tablet fe97e31310ae4733ba93de1b62a37842 (DEFAULT_TABLE table=default.Table_b7db01944072476f9c48985f48b2be65 [id=1ac4b7fed3f145fc9ef5268266aeb9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.185510 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe97e31310ae4733ba93de1b62a37842. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.185539 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c772dc06afa04976b56055df70060bb3" } new_table_name: "default.Table_52ec55c23c394294a68850ab6b33aabe" modify_external_catalogs: true
I20251024 19:47:57.185894 1482 tablet_service.cc:1505] Processing CreateTablet for tablet f05ad092936b4bafa3ae795904c2df78 (DEFAULT_TABLE table=default.Table_99584fc98c7e4f0f853914cc3dda2b12 [id=0704c52aae2c421789a94e4391b9ed2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.186309 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05ad092936b4bafa3ae795904c2df78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.189785 1612 tablet_service.cc:1505] Processing CreateTablet for tablet fe97e31310ae4733ba93de1b62a37842 (DEFAULT_TABLE table=default.Table_b7db01944072476f9c48985f48b2be65 [id=1ac4b7fed3f145fc9ef5268266aeb9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.189860 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 369cebfda6ee43638b79f8652fa55855 (DEFAULT_TABLE table=default.Table_c3cbf88afafa464099c69d8e017743e8 [id=27d44c1a6a2046de8d4e634e4eb6eb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.190174 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 369cebfda6ee43638b79f8652fa55855. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.190225 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe97e31310ae4733ba93de1b62a37842. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.194007 1616 tablet_service.cc:1505] Processing CreateTablet for tablet c292ce055def4964ab143252c6b2b87e (DEFAULT_TABLE table=default.Table_446bc99dda0d455c9362702ac0c2c685 [id=10833938cf88466b834698420d7c01e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.194456 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c292ce055def4964ab143252c6b2b87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.197863 1617 tablet_service.cc:1505] Processing CreateTablet for tablet f05ad092936b4bafa3ae795904c2df78 (DEFAULT_TABLE table=default.Table_99584fc98c7e4f0f853914cc3dda2b12 [id=0704c52aae2c421789a94e4391b9ed2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.198227 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05ad092936b4bafa3ae795904c2df78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.201942 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 369cebfda6ee43638b79f8652fa55855 (DEFAULT_TABLE table=default.Table_c3cbf88afafa464099c69d8e017743e8 [id=27d44c1a6a2046de8d4e634e4eb6eb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.202296 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 369cebfda6ee43638b79f8652fa55855. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.208142 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 90405a0c13bd4c229b011c90232f751d (DEFAULT_TABLE table=default.Table_eb9c8019451d46fba31a87a0b6ac6ad8 [id=e5b1dd31c2114e8a95ba33981029255c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.208631 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90405a0c13bd4c229b011c90232f751d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.210057 1775 ts_tablet_manager.cc:1929] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.210377 1614 tablet_service.cc:1505] Processing CreateTablet for tablet b9b1e0338fd14634a1e921735ae61289 (DEFAULT_TABLE table=default.Table_f02338f14f2b44b3b89757b7edc59e32 [id=3e4514469cd64a8b86d9e44608c186ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.210876 1775 log.cc:1199] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/ae86e74f810c4b989ac633f59e6ac445
I20251024 19:47:57.210904 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9b1e0338fd14634a1e921735ae61289. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.211344 1775 ts_tablet_manager.cc:1950] T ae86e74f810c4b989ac633f59e6ac445 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:57.210675 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 90405a0c13bd4c229b011c90232f751d (DEFAULT_TABLE table=default.Table_eb9c8019451d46fba31a87a0b6ac6ad8 [id=e5b1dd31c2114e8a95ba33981029255c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.211782 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90405a0c13bd4c229b011c90232f751d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.214720 1491 tablet_service.cc:1505] Processing CreateTablet for tablet b9b1e0338fd14634a1e921735ae61289 (DEFAULT_TABLE table=default.Table_f02338f14f2b44b3b89757b7edc59e32 [id=3e4514469cd64a8b86d9e44608c186ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.215224 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9b1e0338fd14634a1e921735ae61289. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.233192 1767 tablet_bootstrap.cc:492] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:57.233444 1767 ts_tablet_manager.cc:1403] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.076s user 0.020s sys 0.036s
I20251024 19:47:57.234616 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet ae86e74f810c4b989ac633f59e6ac445 (table default.Table_ea60f0e4363c406d9ccff5f16b8ecce9 [id=071606963978424489f30719fa42baa3]) successfully deleted
I20251024 19:47:57.234598 1767 raft_consensus.cc:359] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.234998 1767 raft_consensus.cc:385] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.235121 1767 raft_consensus.cc:740] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.235437 1767 consensus_queue.cc:260] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:47:57.235599 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 15c08dcf379f4df9a0e461d6dcfeeac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15c08dcf379f4df9a0e461d6dcfeeac4 already in progress: creating tablet
I20251024 19:47:57.236006 1775 tablet_replica.cc:333] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:57.236436 1775 raft_consensus.cc:2243] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:57.236819 1775 raft_consensus.cc:2272] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.238585 1775 ts_tablet_manager.cc:1916] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.239425 1767 ts_tablet_manager.cc:1434] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251024 19:47:57.242276 1767 tablet_bootstrap.cc:492] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:57.242689 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 9623d7baf0174a019e2353a5effef081 (DEFAULT_TABLE table=default.Table_e629fad818da4d4aa6cf7b4d30895738 [id=281060b712614bbeb46a006b6e34a91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.243561 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 9623d7baf0174a019e2353a5effef081 (DEFAULT_TABLE table=default.Table_e629fad818da4d4aa6cf7b4d30895738 [id=281060b712614bbeb46a006b6e34a91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.243568 1623 tablet_service.cc:1505] Processing CreateTablet for tablet e20881e400cf4fac914b70afc79c084f (DEFAULT_TABLE table=default.Table_69e288a9d6d24a5f841f62cd9a208f0f [id=969d428d45db42d18a5a9b82ec945bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.244148 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20881e400cf4fac914b70afc79c084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.244484 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9623d7baf0174a019e2353a5effef081. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.244489 1767 tablet_bootstrap.cc:654] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.246006 1772 raft_consensus.cc:493] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:57.246188 1772 raft_consensus.cc:515] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.246963 1768 ts_tablet_manager.cc:1434] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251024 19:47:57.248183 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb08e56db59b4db796c50db5bda93af9" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:47:57.249089 1177 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.253:41375, user_credentials={real_user=slave}} blocked reactor thread for 81270.8us
I20251024 19:47:57.249181 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 31647a48fb834c80b6920c756a94250d (DEFAULT_TABLE table=default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.243069 1491 tablet_service.cc:1505] Processing CreateTablet for tablet e20881e400cf4fac914b70afc79c084f (DEFAULT_TABLE table=default.Table_69e288a9d6d24a5f841f62cd9a208f0f [id=969d428d45db42d18a5a9b82ec945bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.252941 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 897db285f5c84930a5b438be1cb4eb96 (DEFAULT_TABLE table=default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.253275 1768 tablet_bootstrap.cc:492] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:57.255812 1768 tablet_bootstrap.cc:654] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.257718 1489 tablet_service.cc:1505] Processing CreateTablet for tablet c04082bd1b444b468c177bef239701c6 (DEFAULT_TABLE table=default.Table_5760bec3ecb94c9c94b3fd2a7ecdd4fd [id=a2789faf4abe4c09b0a7515d941831a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.167671 1865 raft_consensus.cc:515] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:47:57.258718 1308 leader_election.cc:343] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.262230 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 346383e85cc64b3a8da70a71868ceccc (DEFAULT_TABLE table=default.Table_0bc77b2f9654428484956c26da42bf54 [id=44c9c172ae1242d3a7ac272857209ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.262270 1346 tablet_service.cc:1505] Processing CreateTablet for tablet fe97e31310ae4733ba93de1b62a37842 (DEFAULT_TABLE table=default.Table_b7db01944072476f9c48985f48b2be65 [id=1ac4b7fed3f145fc9ef5268266aeb9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.262722 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe97e31310ae4733ba93de1b62a37842. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.262889 1342 tablet_service.cc:1505] Processing CreateTablet for tablet c292ce055def4964ab143252c6b2b87e (DEFAULT_TABLE table=default.Table_446bc99dda0d455c9362702ac0c2c685 [id=10833938cf88466b834698420d7c01e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.263366 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c292ce055def4964ab143252c6b2b87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.263622 1766 tablet_bootstrap.cc:492] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:57.264256 1766 ts_tablet_manager.cc:1403] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.120s user 0.002s sys 0.006s
I20251024 19:47:57.265192 1766 raft_consensus.cc:359] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.265424 1766 raft_consensus.cc:385] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.265522 1766 raft_consensus.cc:740] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.266100 1772 leader_election.cc:290] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:57.266175 1766 consensus_queue.cc:260] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.268256 1350 tablet_service.cc:1505] Processing CreateTablet for tablet f05ad092936b4bafa3ae795904c2df78 (DEFAULT_TABLE table=default.Table_99584fc98c7e4f0f853914cc3dda2b12 [id=0704c52aae2c421789a94e4391b9ed2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.268694 1766 ts_tablet_manager.cc:1434] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251024 19:47:57.269330 1766 tablet_bootstrap.cc:492] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:57.269538 1772 raft_consensus.cc:493] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:57.271225 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 2346c03802a245a2af8c086b626d660d (DEFAULT_TABLE table=default.Table_7c34d3a8affd4e69a31b276ccbe096ae [id=aadca5a1b6d4494aa37f8a8cc76ad12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.274528 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 9623d7baf0174a019e2353a5effef081 (DEFAULT_TABLE table=default.Table_e629fad818da4d4aa6cf7b4d30895738 [id=281060b712614bbeb46a006b6e34a91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.275748 1772 raft_consensus.cc:515] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.276531 1772 leader_election.cc:290] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:57.276780 1865 leader_election.cc:290] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:57.276834 1349 tablet_service.cc:1505] Processing CreateTablet for tablet b9b1e0338fd14634a1e921735ae61289 (DEFAULT_TABLE table=default.Table_f02338f14f2b44b3b89757b7edc59e32 [id=3e4514469cd64a8b86d9e44608c186ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.277654 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 31647a48fb834c80b6920c756a94250d (DEFAULT_TABLE table=default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.277750 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 897db285f5c84930a5b438be1cb4eb96 (DEFAULT_TABLE table=default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:47:57.278220 1447 leader_election.cc:343] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 889bd73511b945b4a4e751f60929a5dc
I20251024 19:47:57.278537 1447 leader_election.cc:304] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:47:57.278915 1447 raft_consensus.cc:2686] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251024 19:47:57.279309 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05ad092936b4bafa3ae795904c2df78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.279345 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 90405a0c13bd4c229b011c90232f751d (DEFAULT_TABLE table=default.Table_eb9c8019451d46fba31a87a0b6ac6ad8 [id=e5b1dd31c2114e8a95ba33981029255c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.283056 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 369cebfda6ee43638b79f8652fa55855 (DEFAULT_TABLE table=default.Table_c3cbf88afafa464099c69d8e017743e8 [id=27d44c1a6a2046de8d4e634e4eb6eb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.279310 1344 tablet_service.cc:1505] Processing CreateTablet for tablet e20881e400cf4fac914b70afc79c084f (DEFAULT_TABLE table=default.Table_69e288a9d6d24a5f841f62cd9a208f0f [id=969d428d45db42d18a5a9b82ec945bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.289279 1351 tablet_service.cc:1505] Processing CreateTablet for tablet c04082bd1b444b468c177bef239701c6 (DEFAULT_TABLE table=default.Table_5760bec3ecb94c9c94b3fd2a7ecdd4fd [id=a2789faf4abe4c09b0a7515d941831a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.289958 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31647a48fb834c80b6920c756a94250d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:57.291548 1308 leader_election.cc:343] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: ae86e74f810c4b989ac633f59e6ac445
I20251024 19:47:57.292572 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb17a35d8d8446cf9870d05e107ed237" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:57.293886 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9623d7baf0174a019e2353a5effef081. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.293934 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 2346c03802a245a2af8c086b626d660d (DEFAULT_TABLE table=default.Table_7c34d3a8affd4e69a31b276ccbe096ae [id=aadca5a1b6d4494aa37f8a8cc76ad12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.294409 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20881e400cf4fac914b70afc79c084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.294404 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 346383e85cc64b3a8da70a71868ceccc (DEFAULT_TABLE table=default.Table_0bc77b2f9654428484956c26da42bf54 [id=44c9c172ae1242d3a7ac272857209ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.294986 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 346383e85cc64b3a8da70a71868ceccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.296783 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 369cebfda6ee43638b79f8652fa55855. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.297530 1865 raft_consensus.cc:493] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251024 19:47:57.296396 1308 leader_election.cc:343] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.298202 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2346c03802a245a2af8c086b626d660d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.298203 1865 raft_consensus.cc:515] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:57.298560 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04082bd1b444b468c177bef239701c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.299000 1865 leader_election.cc:290] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:57.300288 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f21ffa44d7f48d08ed17ebf6c8ff29c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:57.301203 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90405a0c13bd4c229b011c90232f751d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.302567 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897db285f5c84930a5b438be1cb4eb96. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:57.303763 1308 leader_election.cc:343] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.305155 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9b1e0338fd14634a1e921735ae61289. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.306221 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 31647a48fb834c80b6920c756a94250d (DEFAULT_TABLE table=default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.306922 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31647a48fb834c80b6920c756a94250d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.310269 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb08e56db59b4db796c50db5bda93af9" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:57.310765 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae86e74f810c4b989ac633f59e6ac445" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:57.311120 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb17a35d8d8446cf9870d05e107ed237" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:57.311781 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f21ffa44d7f48d08ed17ebf6c8ff29c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:57.313042 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9623d7baf0174a019e2353a5effef081. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.313334 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 897db285f5c84930a5b438be1cb4eb96 (DEFAULT_TABLE table=default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.313449 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2346c03802a245a2af8c086b626d660d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.313817 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897db285f5c84930a5b438be1cb4eb96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.317727 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897db285f5c84930a5b438be1cb4eb96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.318447 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 346383e85cc64b3a8da70a71868ceccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.322027 1615 tablet_service.cc:1505] Processing CreateTablet for tablet c04082bd1b444b468c177bef239701c6 (DEFAULT_TABLE table=default.Table_5760bec3ecb94c9c94b3fd2a7ecdd4fd [id=a2789faf4abe4c09b0a7515d941831a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.322536 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04082bd1b444b468c177bef239701c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.325942 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 346383e85cc64b3a8da70a71868ceccc (DEFAULT_TABLE table=default.Table_0bc77b2f9654428484956c26da42bf54 [id=44c9c172ae1242d3a7ac272857209ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.326318 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 346383e85cc64b3a8da70a71868ceccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.329979 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 2346c03802a245a2af8c086b626d660d (DEFAULT_TABLE table=default.Table_7c34d3a8affd4e69a31b276ccbe096ae [id=aadca5a1b6d4494aa37f8a8cc76ad12a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.330358 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2346c03802a245a2af8c086b626d660d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.335314 1766 tablet_bootstrap.cc:654] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.336470 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04082bd1b444b468c177bef239701c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.341737 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31647a48fb834c80b6920c756a94250d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.343314 1775 ts_tablet_manager.cc:1929] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.344178 1775 log.cc:1199] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/889bd73511b945b4a4e751f60929a5dc
I20251024 19:47:57.346196 1186 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 922f975a672746b3b72acf4d3050f688 (table default.Table_ab42d318a6cc4626be3d62dda5948bf0 [id=eab377d5ce1343d7a12c3740fe46beb5]). Expected version 3 got 0
I20251024 19:47:57.350179 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_446bc99dda0d455c9362702ac0c2c685" } new_table_name: "default.Table_5e63f20b2f66459795a5289f47ffaf0e" modify_external_catalogs: true
W20251024 19:47:57.352072 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 979dd87a8c66480ea314fcc75fa878c3 consensus state has no leader
I20251024 19:47:57.353410 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f683ab67c4774e25861c7821970d1981" } new_table_name: "default.Table_51982d999b0847ec87fc62a473782a5c" modify_external_catalogs: true
I20251024 19:47:57.353859 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c4539f71581f488cad3bc3b4b2f3a0eb" } new_table_name: "default.Table_a54ffafa18b848b79a290f58c9754bec" modify_external_catalogs: true
I20251024 19:47:57.354410 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20881e400cf4fac914b70afc79c084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.354883 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 110907ae943f45b5bcaa9b66ccdb84fd
I20251024 19:47:57.356097 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_656be68a26b545d2b254abc2edb15254" } modify_external_catalogs: true
I20251024 19:47:57.356259 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_656be68a26b545d2b254abc2edb15254" } modify_external_catalogs: true
I20251024 19:47:57.356932 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ffaa4997edd4b1187579958037ea793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.357156 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 110907ae943f45b5bcaa9b66ccdb84fd with delete_type TABLET_DATA_DELETED (replaced by tablet 1d04a9b54bd6446d8deb64292cef672d) from {username='slave'} at 127.0.0.1:46462
W20251024 19:47:57.357728 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95d7fbfa8ae54da49dd5603294faa7d3 consensus state has no leader
I20251024 19:47:57.359136 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet 2d941eafb1414f068dc656946082f617 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:57.359874 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_272323b2c4ac4c8cb0ff412a632ee43d" } modify_external_catalogs: true
I20251024 19:47:57.360073 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_272323b2c4ac4c8cb0ff412a632ee43d" } modify_external_catalogs: true
I20251024 19:47:57.361689 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 95d7fbfa8ae54da49dd5603294faa7d3
I20251024 19:47:57.362401 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f05ad092936b4bafa3ae795904c2df78
I20251024 19:47:57.369755 1766 tablet_bootstrap.cc:492] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:57.371320 1775 ts_tablet_manager.cc:1950] T 889bd73511b945b4a4e751f60929a5dc P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:57.371410 1766 ts_tablet_manager.cc:1403] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.102s user 0.006s sys 0.002s
I20251024 19:47:57.373757 1766 raft_consensus.cc:359] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.374081 1766 raft_consensus.cc:385] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.373759 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet f05ad092936b4bafa3ae795904c2df78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.374411 1766 raft_consensus.cc:740] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.374752 1766 consensus_queue.cc:260] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.376026 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 31647a48fb834c80b6920c756a94250d
I20251024 19:47:57.380700 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 95d7fbfa8ae54da49dd5603294faa7d3 with delete_type TABLET_DATA_DELETED (replaced by tablet dc1b6f0a377945089c50012f74d38564) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.381079 1766 ts_tablet_manager.cc:1434] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:47:57.382027 1766 tablet_bootstrap.cc:492] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:57.386123 1766 tablet_bootstrap.cc:654] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:47:57.391274 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1ad65fd080e452e8094975d420336c7 consensus state has no leader
I20251024 19:47:57.392467 1184 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 922f975a672746b3b72acf4d3050f688 (table default.Table_ab42d318a6cc4626be3d62dda5948bf0 [id=eab377d5ce1343d7a12c3740fe46beb5]). Expected version 3 got 0
I20251024 19:47:57.393290 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ab42d318a6cc4626be3d62dda5948bf0" } modify_external_catalogs: true
I20251024 19:47:57.399434 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ab42d318a6cc4626be3d62dda5948bf0" } modify_external_catalogs: true
W20251024 19:47:57.399950 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb17a35d8d8446cf9870d05e107ed237 consensus state has no leader
I20251024 19:47:57.400848 1190 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 922f975a672746b3b72acf4d3050f688 (table default.Table_ab42d318a6cc4626be3d62dda5948bf0 [id=eab377d5ce1343d7a12c3740fe46beb5]). Expected version 3 got 0
W20251024 19:47:57.402163 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c292ce055def4964ab143252c6b2b87e consensus state has no leader
I20251024 19:47:57.402550 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4db7a2da5af4481db5b546d201dfd278"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:57.403452 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c
I20251024 19:47:57.404559 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_32d4a5b034974da0a745844ffcab9498"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:57.404742 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.407224 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet cc0bf7aee1a1421b9d985020a1674639
I20251024 19:47:57.407619 1766 tablet_bootstrap.cc:492] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:57.407883 1766 ts_tablet_manager.cc:1403] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.001s
I20251024 19:47:57.408762 1766 raft_consensus.cc:359] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.408928 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet cc0bf7aee1a1421b9d985020a1674639 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.408983 1766 raft_consensus.cc:385] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.409056 1766 raft_consensus.cc:740] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.409298 1766 consensus_queue.cc:260] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.409934 1766 ts_tablet_manager.cc:1434] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251024 19:47:57.410241 1766 tablet_bootstrap.cc:492] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:57.412206 1766 tablet_bootstrap.cc:654] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.415643 1185 catalog_manager.cc:6719] ReplaceTablet: tablet 95d7fbfa8ae54da49dd5603294faa7d3 deleted and replaced by tablet dc1b6f0a377945089c50012f74d38564
I20251024 19:47:57.416147 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_edfd7941851340ecbcd71d4b48ff583f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.416756 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edfd7941851340ecbcd71d4b48ff583f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.417630 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ff7a43292b9045f9bbfa71883bcd7e61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.418262 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff7a43292b9045f9bbfa71883bcd7e61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.418466 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1142a96660b74e09be367f32b979af21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.418911 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1142a96660b74e09be367f32b979af21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.415678 1191 catalog_manager.cc:6719] ReplaceTablet: tablet 110907ae943f45b5bcaa9b66ccdb84fd deleted and replaced by tablet 1d04a9b54bd6446d8deb64292cef672d
I20251024 19:47:57.420317 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b8e34cb629054ed9be6deaf4d30992a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.420534 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4db7a2da5af4481db5b546d201dfd278 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:57.420895 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8e34cb629054ed9be6deaf4d30992a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:57.427052 1179 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet c04082bd1b444b468c177bef239701c6 (table default.Table_5760bec3ecb94c9c94b3fd2a7ecdd4fd [id=a2789faf4abe4c09b0a7515d941831a6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c04082bd1b444b468c177bef239701c6 consensus state has no leader
I20251024 19:47:57.428916 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b2f85cccb274d9da74bb37f91f020af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.429057 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 31647a48fb834c80b6920c756a94250d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:47:57.439388 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32d4a5b034974da0a745844ffcab9498 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.446975 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 889bd73511b945b4a4e751f60929a5dc (table default.Table_a2fc8ac42bbc44af8bf194f2d3511bd5 [id=5518afdf09da4433bee8f2221ead84c1]) successfully deleted
I20251024 19:47:57.460898 1186 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]). Expected version 3 got 0
W20251024 19:47:57.462143 1186 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet fb08e56db59b4db796c50db5bda93af9 consensus state has no leader
I20251024 19:47:57.473186 1775 tablet_replica.cc:333] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:57.473544 1775 raft_consensus.cc:2243] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:57.475049 1775 raft_consensus.cc:2272] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.476050 1775 ts_tablet_manager.cc:1916] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.480338 1782 raft_consensus.cc:493] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:57.482036 1865 raft_consensus.cc:493] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:57.482172 1774 ts_tablet_manager.cc:1929] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.482223 1865 raft_consensus.cc:515] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:57.482391 1774 log.cc:1199] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/e2d97caee586418a8981108465f6b704
I20251024 19:47:57.483038 1774 ts_tablet_manager.cc:1950] T e2d97caee586418a8981108465f6b704 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:57.483217 1489 tablet_service.cc:1505] Processing CreateTablet for tablet a48249f1824a40b087b80a563816d485 (DEFAULT_TABLE table=default.Table_f96cb6877d844a50aacb91514a539fee [id=d9ab6489833b4989a69c975d734bcd2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.483502 1865 leader_election.cc:290] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:57.484655 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48249f1824a40b087b80a563816d485. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.484560 1782 raft_consensus.cc:515] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:57.485234 1782 leader_election.cc:290] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:47:57.485363 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b2f85cccb274d9da74bb37f91f020af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.489720 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc8c01a4c07a42a0bc48a58313090e68" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:57.490509 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 2496f103436047dea5dabb577651249f (DEFAULT_TABLE table=default.Table_66accd4024cc4cd98f10a25df857d785 [id=4e52676da82e4f5f807ffc32fb747281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.490922 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2496f103436047dea5dabb577651249f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:57.492202 1305 leader_election.cc:343] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.493409 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1ad65fd080e452e8094975d420336c7" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:57.493806 1511 raft_consensus.cc:2468] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:47:57.495246 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 922f975a672746b3b72acf4d3050f688
I20251024 19:47:57.502051 1305 leader_election.cc:304] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:57.502628 1586 leader_election.cc:304] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:47:57.503376 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 110907ae943f45b5bcaa9b66ccdb84fd with delete_type TABLET_DATA_DELETED (replaced by tablet 1d04a9b54bd6446d8deb64292cef672d) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.503666 1483 tablet_service.cc:1505] Processing CreateTablet for tablet f346a90a9efe442f82750336e3893deb (DEFAULT_TABLE table=default.Table_673f2f271b3e4f9d9c5f6802a3646f8f [id=b79808d531b740d7abc0e6fc2d9ba219]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.503891 1776 tablet_replica.cc:333] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:57.504199 1782 raft_consensus.cc:2804] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:47:57.504287 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet f05ad092936b4bafa3ae795904c2df78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.504312 1782 raft_consensus.cc:493] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:47:57.504915 1782 raft_consensus.cc:3060] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:47:57.505163 1305 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 180438us
W20251024 19:47:57.505527 1305 leader_election.cc:343] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.505722 1305 leader_election.cc:304] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:57.506067 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 95d7fbfa8ae54da49dd5603294faa7d3 with delete_type TABLET_DATA_DELETED (replaced by tablet dc1b6f0a377945089c50012f74d38564) from {username='slave'} at 127.0.0.1:54246
W20251024 19:47:57.506139 1305 leader_election.cc:343] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.506304 1305 leader_election.cc:304] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
W20251024 19:47:57.506665 1305 leader_election.cc:343] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.506656 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 889bd73511b945b4a4e751f60929a5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.506798 1305 leader_election.cc:304] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:57.506479 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 922f975a672746b3b72acf4d3050f688 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.507169 1772 raft_consensus.cc:2749] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:57.507205 1776 raft_consensus.cc:2243] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:57.507373 1772 raft_consensus.cc:2749] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:57.507408 1776 raft_consensus.cc:2272] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.507336 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 7c357a2a81064b3891afda6d252898b9 (DEFAULT_TABLE table=default.Table_2e71745e53a04c5b8701ff72aedb203c [id=ed879934f5754b92b8cff234f3f66333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.507550 1772 raft_consensus.cc:2749] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:57.507746 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f346a90a9efe442f82750336e3893deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.508330 1491 tablet_service.cc:1505] Processing CreateTablet for tablet b5473699c1b647f8b86ee43fdb56bdfd (DEFAULT_TABLE table=default.Table_c700d3a5bb0c4963ace8624ae2ef882d [id=30691828be584171b7ee2472c9a25b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.508913 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c357a2a81064b3891afda6d252898b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.508948 1776 ts_tablet_manager.cc:1916] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.508977 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:47:57.509804 1308 leader_election.cc:343] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.509871 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5473699c1b647f8b86ee43fdb56bdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.511047 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet cc0bf7aee1a1421b9d985020a1674639 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.511716 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 31647a48fb834c80b6920c756a94250d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.513751 1615 tablet_service.cc:1505] Processing CreateTablet for tablet a48249f1824a40b087b80a563816d485 (DEFAULT_TABLE table=default.Table_f96cb6877d844a50aacb91514a539fee [id=d9ab6489833b4989a69c975d734bcd2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.514559 1865 raft_consensus.cc:2749] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:57.514616 1618 tablet_service.cc:1505] Processing CreateTablet for tablet b5473699c1b647f8b86ee43fdb56bdfd (DEFAULT_TABLE table=default.Table_c700d3a5bb0c4963ace8624ae2ef882d [id=30691828be584171b7ee2472c9a25b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.515929 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 2496f103436047dea5dabb577651249f (DEFAULT_TABLE table=default.Table_66accd4024cc4cd98f10a25df857d785 [id=4e52676da82e4f5f807ffc32fb747281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.516381 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1ad65fd080e452e8094975d420336c7" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:47:57.516650 1371 raft_consensus.cc:2468] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:47:57.517614 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 7d1933bb728845abad5e6b09b24868f0 (DEFAULT_TABLE table=default.Table_c564241bf74c49deaa26052b5c1d5afe [id=ec0b3dc60c1a47f0b77f5de634de0b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.524667 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 7c357a2a81064b3891afda6d252898b9 (DEFAULT_TABLE table=default.Table_2e71745e53a04c5b8701ff72aedb203c [id=ed879934f5754b92b8cff234f3f66333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.525565 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 7d1933bb728845abad5e6b09b24868f0 (DEFAULT_TABLE table=default.Table_c564241bf74c49deaa26052b5c1d5afe [id=ec0b3dc60c1a47f0b77f5de634de0b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.527658 1782 raft_consensus.cc:515] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:57.529713 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d1933bb728845abad5e6b09b24868f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.530184 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48249f1824a40b087b80a563816d485. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.530524 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1ad65fd080e452e8094975d420336c7" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:47:57.530727 1511 raft_consensus.cc:3060] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:57.532290 1782 leader_election.cc:290] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:47:57.532928 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 922f975a672746b3b72acf4d3050f688 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.533607 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2496f103436047dea5dabb577651249f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.533836 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5473699c1b647f8b86ee43fdb56bdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.534024 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c357a2a81064b3891afda6d252898b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.535842 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d1933bb728845abad5e6b09b24868f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.541922 1774 tablet_replica.cc:333] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:57.542187 1774 raft_consensus.cc:2243] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:57.542379 1774 raft_consensus.cc:2272] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.542995 1774 ts_tablet_manager.cc:1916] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.547246 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_999546c6f492460bb6dc96ab55b7d3ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.547911 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_999546c6f492460bb6dc96ab55b7d3ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.551937 1775 ts_tablet_manager.cc:1929] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.552126 1775 log.cc:1199] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/0ffaa4997edd4b1187579958037ea793
I20251024 19:47:57.552589 1775 ts_tablet_manager.cc:1950] T 0ffaa4997edd4b1187579958037ea793 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:57.553128 1614 tablet_service.cc:1505] Processing CreateTablet for tablet f346a90a9efe442f82750336e3893deb (DEFAULT_TABLE table=default.Table_673f2f271b3e4f9d9c5f6802a3646f8f [id=b79808d531b740d7abc0e6fc2d9ba219]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.553783 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 0ffaa4997edd4b1187579958037ea793 (table default.Table_cf69470650f643c195eb5fbecbd2d2c4 [id=c1c8743e95d34655b1a8779920ed837a]) successfully deleted
I20251024 19:47:57.555130 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_cabda156901b4ee38d0be4dd4efe8376"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.555353 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 110907ae943f45b5bcaa9b66ccdb84fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110907ae943f45b5bcaa9b66ccdb84fd already in progress: creating tablet
W20251024 19:47:57.555755 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cabda156901b4ee38d0be4dd4efe8376 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:57.556628 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet f05ad092936b4bafa3ae795904c2df78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f05ad092936b4bafa3ae795904c2df78 already in progress: creating tablet
W20251024 19:47:57.557212 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 95d7fbfa8ae54da49dd5603294faa7d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95d7fbfa8ae54da49dd5603294faa7d3 already in progress: creating tablet
W20251024 19:47:57.557904 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c already in progress: creating tablet
W20251024 19:47:57.558521 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet cc0bf7aee1a1421b9d985020a1674639 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc0bf7aee1a1421b9d985020a1674639 already in progress: creating tablet
I20251024 19:47:57.558801 1776 ts_tablet_manager.cc:1929] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.558939 1775 tablet_replica.cc:333] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:57.559175 1775 raft_consensus.cc:2243] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:57.559398 1775 raft_consensus.cc:2272] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.560228 1775 ts_tablet_manager.cc:1916] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.560279 1776 log.cc:1199] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/8b2f85cccb274d9da74bb37f91f020af
I20251024 19:47:57.560623 1776 ts_tablet_manager.cc:1950] T 8b2f85cccb274d9da74bb37f91f020af P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:57.561664 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 8b2f85cccb274d9da74bb37f91f020af (table default.Table_8d9e77c4fcbf4a6e9f90d1c111a8cb50 [id=9b044de638ad4f14b886d199bc4a2532]) successfully deleted
W20251024 19:47:57.562088 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 110907ae943f45b5bcaa9b66ccdb84fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110907ae943f45b5bcaa9b66ccdb84fd already in progress: creating tablet
W20251024 19:47:57.562376 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f05ad092936b4bafa3ae795904c2df78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f05ad092936b4bafa3ae795904c2df78 already in progress: creating tablet
I20251024 19:47:57.564042 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f346a90a9efe442f82750336e3893deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.565656 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_97c6fc690615425aa5af0d8c0dd249af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.566254 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97c6fc690615425aa5af0d8c0dd249af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.566388 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9df1d5f7c43c450b9a68df4d77f6ab53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.566876 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9df1d5f7c43c450b9a68df4d77f6ab53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.567463 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0bc77b2f9654428484956c26da42bf54" } new_table_name: "default.Table_acb41c25b40742ef92f4bb9d45ada0a3" modify_external_catalogs: true
W20251024 19:47:57.568291 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 95d7fbfa8ae54da49dd5603294faa7d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95d7fbfa8ae54da49dd5603294faa7d3 already in progress: creating tablet
I20251024 19:47:57.570461 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4dda32c7ff144630bebd572cf5f9f822" } new_table_name: "default.Table_d5ca9b9774254589b656ade4f08dc4c1" modify_external_catalogs: true
I20251024 19:47:57.575103 1865 raft_consensus.cc:493] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:57.575323 1865 raft_consensus.cc:515] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.577281 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "608696ec90ed45e1ad939ed2bfc88d34" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:57.580758 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1ad65fd080e452e8094975d420336c7" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:47:57.581748 1189 master_service.cc:954] ReplaceTablet: received request to replace tablet f396c2982f8d487cb44c9422b09da14a from {username='slave'} at 127.0.0.1:51416
W20251024 19:47:57.587062 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 346383e85cc64b3a8da70a71868ceccc consensus state has no leader
W20251024 19:47:57.593873 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e956ea914c545d79be6d632bfc4ec57 consensus state has no leader
I20251024 19:47:57.598829 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9df1d5f7c43c450b9a68df4d77f6ab53" } new_table_name: "default.Table_52ecaf891a2449d78bb8a354e7fb904b" modify_external_catalogs: true
W20251024 19:47:57.599310 1308 leader_election.cc:343] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.602505 1865 leader_election.cc:290] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:57.602632 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b7db01944072476f9c48985f48b2be65" } new_table_name: "default.Table_450ee0c26fcb4ab7961f7cbde38becb1" modify_external_catalogs: true
I20251024 19:47:57.602777 1371 raft_consensus.cc:3060] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:57.604820 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e629fad818da4d4aa6cf7b4d30895738" } new_table_name: "default.Table_7f14fdc529ec48d68d03ca041f64a53f" modify_external_catalogs: true
W20251024 19:47:57.611991 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe97e31310ae4733ba93de1b62a37842 consensus state has no leader
W20251024 19:47:57.612613 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ec5c7bf36874423b4790c6169da8bd0 has no consensus state
W20251024 19:47:57.613272 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9623d7baf0174a019e2353a5effef081 consensus state has no leader
I20251024 19:47:57.618561 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 6a7e76a3cac443ec8d2ab466075aae6b (DEFAULT_TABLE table=default.Table_32d4a5b034974da0a745844ffcab9498 [id=5fc5a169166b436ba9343a652a650b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.619061 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a7e76a3cac443ec8d2ab466075aae6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.620322 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 913534ab88dd459e9cac2faebb5492a1 (DEFAULT_TABLE table=default.Table_edfd7941851340ecbcd71d4b48ff583f [id=acce7ccf5ddb40efab47c44305393922]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.620766 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913534ab88dd459e9cac2faebb5492a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.623188 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 50d919a0b3d74e9a97d136a84d50e24b (DEFAULT_TABLE table=default.Table_b9caeb0e904f44dc8f879c8988a34678 [id=6652d9398f8e4495bdfbca917b1d0bab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.623593 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50d919a0b3d74e9a97d136a84d50e24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.627401 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 857eed60ba2a4b309e86d88a0f093762 (DEFAULT_TABLE table=default.Table_b8e34cb629054ed9be6deaf4d30992a2 [id=d225099585ad4533b088821c639b5463]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.629346 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6e1fbc007b464e8c93803ba262a19fea" } modify_external_catalogs: true
I20251024 19:47:57.629616 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6e1fbc007b464e8c93803ba262a19fea" } modify_external_catalogs: true
I20251024 19:47:57.629683 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f02338f14f2b44b3b89757b7edc59e32" } modify_external_catalogs: true
I20251024 19:47:57.629935 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f02338f14f2b44b3b89757b7edc59e32" } modify_external_catalogs: true
I20251024 19:47:57.637264 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_46b7ffb5bf6f45f2be4d1b242f43bfc9" } modify_external_catalogs: true
I20251024 19:47:57.637482 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_46b7ffb5bf6f45f2be4d1b242f43bfc9" } modify_external_catalogs: true
I20251024 19:47:57.642706 1776 tablet_replica.cc:333] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:57.644459 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ce25ea67811e4818be50de93930ab68e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.645385 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce25ea67811e4818be50de93930ab68e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.645673 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 913534ab88dd459e9cac2faebb5492a1 (DEFAULT_TABLE table=default.Table_edfd7941851340ecbcd71d4b48ff583f [id=acce7ccf5ddb40efab47c44305393922]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.646057 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913534ab88dd459e9cac2faebb5492a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.648121 1371 raft_consensus.cc:2468] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:47:57.649384 1586 leader_election.cc:304] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:47:57.649848 1782 raft_consensus.cc:2804] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:47:57.651153 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ebef5e3474bb4a288fcbaf6def6a7ce1" } modify_external_catalogs: true
I20251024 19:47:57.651420 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ebef5e3474bb4a288fcbaf6def6a7ce1" } modify_external_catalogs: true
I20251024 19:47:57.653167 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5a1d470fc7bc493393e47da7db62dfca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:57.654482 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f46ed7e555b3432b95a3d9b829968b8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.655187 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f46ed7e555b3432b95a3d9b829968b8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.656575 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet 65f646bd05274cb6859c0b7f92b61352 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:57.656571 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 6a7e76a3cac443ec8d2ab466075aae6b (DEFAULT_TABLE table=default.Table_32d4a5b034974da0a745844ffcab9498 [id=5fc5a169166b436ba9343a652a650b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.657642 1776 raft_consensus.cc:2243] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:47:57.659932 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a1d470fc7bc493393e47da7db62dfca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.660261 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_eb1e63c7ef2244259c8a64fd4fe3859b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.660794 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb1e63c7ef2244259c8a64fd4fe3859b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.661129 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet fb17a35d8d8446cf9870d05e107ed237 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:57.664135 1511 raft_consensus.cc:2468] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:47:57.666464 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 857eed60ba2a4b309e86d88a0f093762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.680298 1768 tablet_bootstrap.cc:492] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:57.680493 1768 ts_tablet_manager.cc:1403] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.433s user 0.020s sys 0.041s
I20251024 19:47:57.681349 1768 raft_consensus.cc:359] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:57.681545 1768 raft_consensus.cc:385] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.681663 1768 raft_consensus.cc:740] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.681902 1768 consensus_queue.cc:260] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:57.684105 1768 ts_tablet_manager.cc:1434] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:47:57.684432 1768 tablet_bootstrap.cc:492] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:57.686283 1768 tablet_bootstrap.cc:654] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.696879 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a7e76a3cac443ec8d2ab466075aae6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.697245 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 50d919a0b3d74e9a97d136a84d50e24b (DEFAULT_TABLE table=default.Table_b9caeb0e904f44dc8f879c8988a34678 [id=6652d9398f8e4495bdfbca917b1d0bab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.698155 1775 ts_tablet_manager.cc:1929] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.698490 1775 log.cc:1199] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/8b2f85cccb274d9da74bb37f91f020af
I20251024 19:47:57.699177 1775 ts_tablet_manager.cc:1950] T 8b2f85cccb274d9da74bb37f91f020af P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:57.700979 1775 tablet_replica.cc:333] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:57.701357 1775 raft_consensus.cc:2243] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:57.701702 1775 raft_consensus.cc:2272] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.686712 1865 raft_consensus.cc:493] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:57.704457 1865 raft_consensus.cc:515] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.705130 1865 leader_election.cc:290] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:57.705169 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 8b2f85cccb274d9da74bb37f91f020af (table default.Table_8d9e77c4fcbf4a6e9f90d1c111a8cb50 [id=9b044de638ad4f14b886d199bc4a2532]) successfully deleted
W20251024 19:47:57.705479 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 31647a48fb834c80b6920c756a94250d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31647a48fb834c80b6920c756a94250d already in progress: creating tablet
I20251024 19:47:57.692724 1774 ts_tablet_manager.cc:1929] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251024 19:47:57.698357 1305 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 191688us
I20251024 19:47:57.710378 1774 log.cc:1199] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/2d941eafb1414f068dc656946082f617
I20251024 19:47:57.710843 1774 ts_tablet_manager.cc:1950] T 2d941eafb1414f068dc656946082f617 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:57.711333 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 95d7fbfa8ae54da49dd5603294faa7d3 with delete_type TABLET_DATA_DELETED (replaced by tablet dc1b6f0a377945089c50012f74d38564) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.711786 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.711865 1774 tablet_replica.cc:333] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:57.712877 1774 raft_consensus.cc:2243] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:57.713372 1774 raft_consensus.cc:2272] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.712551 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 2496f103436047dea5dabb577651249f (DEFAULT_TABLE table=default.Table_66accd4024cc4cd98f10a25df857d785 [id=4e52676da82e4f5f807ffc32fb747281]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.713668 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 857eed60ba2a4b309e86d88a0f093762 (DEFAULT_TABLE table=default.Table_b8e34cb629054ed9be6deaf4d30992a2 [id=d225099585ad4533b088821c639b5463]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.714043 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2496f103436047dea5dabb577651249f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.714087 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet e2d97caee586418a8981108465f6b704 (table default.Table_95644b1e2fa0449d806efb9c458266b9 [id=168a2adf03184a04b7e8f5017919a08c]) successfully deleted
I20251024 19:47:57.714315 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 7c357a2a81064b3891afda6d252898b9 (DEFAULT_TABLE table=default.Table_2e71745e53a04c5b8701ff72aedb203c [id=ed879934f5754b92b8cff234f3f66333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.714704 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c357a2a81064b3891afda6d252898b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.715126 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 2d941eafb1414f068dc656946082f617 (table default.Table_ee4ef17d5f314379846d30a477d4911d [id=a402387542654ccf8e0b04ca8ecebb1a]) successfully deleted
I20251024 19:47:57.715590 1776 raft_consensus.cc:2272] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.717893 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 7d1933bb728845abad5e6b09b24868f0 (DEFAULT_TABLE table=default.Table_c564241bf74c49deaa26052b5c1d5afe [id=ec0b3dc60c1a47f0b77f5de634de0b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.718232 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d1933bb728845abad5e6b09b24868f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.718273 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50d919a0b3d74e9a97d136a84d50e24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.718575 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 857eed60ba2a4b309e86d88a0f093762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.718528 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 913534ab88dd459e9cac2faebb5492a1 (DEFAULT_TABLE table=default.Table_edfd7941851340ecbcd71d4b48ff583f [id=acce7ccf5ddb40efab47c44305393922]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.719748 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 50d919a0b3d74e9a97d136a84d50e24b (DEFAULT_TABLE table=default.Table_b9caeb0e904f44dc8f879c8988a34678 [id=6652d9398f8e4495bdfbca917b1d0bab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.720717 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 6a7e76a3cac443ec8d2ab466075aae6b (DEFAULT_TABLE table=default.Table_32d4a5b034974da0a745844ffcab9498 [id=5fc5a169166b436ba9343a652a650b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.721702 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 857eed60ba2a4b309e86d88a0f093762 (DEFAULT_TABLE table=default.Table_b8e34cb629054ed9be6deaf4d30992a2 [id=d225099585ad4533b088821c639b5463]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.722102 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 857eed60ba2a4b309e86d88a0f093762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.722977 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b9b1e0338fd14634a1e921735ae61289
I20251024 19:47:57.723580 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 65f646bd05274cb6859c0b7f92b61352
I20251024 19:47:57.724026 1183 catalog_manager.cc:6719] ReplaceTablet: tablet 65f646bd05274cb6859c0b7f92b61352 deleted and replaced by tablet aebf76d4a66947ed9756643740418798
I20251024 19:47:57.724009 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 922f975a672746b3b72acf4d3050f688 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.724540 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet 9820e68ad5264df1a79b22f3b5369c5a from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:57.727146 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e9bc5ade2870426abe90a5f3b20ebb5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.728214 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9bc5ade2870426abe90a5f3b20ebb5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.728475 1352 tablet_service.cc:1505] Processing CreateTablet for tablet f346a90a9efe442f82750336e3893deb (DEFAULT_TABLE table=default.Table_673f2f271b3e4f9d9c5f6802a3646f8f [id=b79808d531b740d7abc0e6fc2d9ba219]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.728606 1776 ts_tablet_manager.cc:1916] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.728667 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 8a1f1892bd3b4776adc7c2519fcdf411
I20251024 19:47:57.728838 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet f05ad092936b4bafa3ae795904c2df78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.729106 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913534ab88dd459e9cac2faebb5492a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.729326 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8c169b038d764be684a4e5b56faf745b" } modify_external_catalogs: true
I20251024 19:47:57.729475 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 608696ec90ed45e1ad939ed2bfc88d34
I20251024 19:47:57.730614 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet cc0bf7aee1a1421b9d985020a1674639 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:57.731115 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 230606a0ce084b7bab0e41e48cf8fa89
I20251024 19:47:57.731712 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_573ec53c3d2c4de8864f399b291d8aa9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.732261 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_573ec53c3d2c4de8864f399b291d8aa9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.732591 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet b9b1e0338fd14634a1e921735ae61289 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.732921 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 608696ec90ed45e1ad939ed2bfc88d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.733003 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a7e76a3cac443ec8d2ab466075aae6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.733040 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc8c01a4c07a42a0bc48a58313090e68" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:57.733425 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 65f646bd05274cb6859c0b7f92b61352 with delete_type TABLET_DATA_DELETED (replaced by tablet aebf76d4a66947ed9756643740418798) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.733316 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8c169b038d764be684a4e5b56faf745b" } modify_external_catalogs: true
I20251024 19:47:57.733665 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a1f1892bd3b4776adc7c2519fcdf411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.734274 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet 230606a0ce084b7bab0e41e48cf8fa89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.734776 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "922f975a672746b3b72acf4d3050f688" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:57.735009 1649 raft_consensus.cc:2468] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:47:57.735628 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "608696ec90ed45e1ad939ed2bfc88d34" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:57.736151 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 110907ae943f45b5bcaa9b66ccdb84fd with delete_type TABLET_DATA_DELETED (replaced by tablet 1d04a9b54bd6446d8deb64292cef672d) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.737353 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet cc0bf7aee1a1421b9d985020a1674639 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.738003 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50d919a0b3d74e9a97d136a84d50e24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.738310 1767 tablet_bootstrap.cc:492] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:57.738171 1348 tablet_service.cc:1505] Processing CreateTablet for tablet b5473699c1b647f8b86ee43fdb56bdfd (DEFAULT_TABLE table=default.Table_c700d3a5bb0c4963ace8624ae2ef882d [id=30691828be584171b7ee2472c9a25b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.745806 1767 ts_tablet_manager.cc:1403] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.504s user 0.022s sys 0.045s
I20251024 19:47:57.705741 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "922f975a672746b3b72acf4d3050f688" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:57.747936 1767 raft_consensus.cc:359] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.748128 1767 raft_consensus.cc:385] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.750564 1767 raft_consensus.cc:740] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.751128 1767 consensus_queue.cc:260] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.752282 1884 raft_consensus.cc:493] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:57.752655 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9820e68ad5264df1a79b22f3b5369c5a
I20251024 19:47:57.753161 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet fb17a35d8d8446cf9870d05e107ed237
I20251024 19:47:57.753708 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3caecdff61ce48f7b7c3846bb380d21e
I20251024 19:47:57.753772 1187 catalog_manager.cc:6719] ReplaceTablet: tablet fb17a35d8d8446cf9870d05e107ed237 deleted and replaced by tablet ca0527af718f494a8bbcc66e7b325bed
I20251024 19:47:57.754065 1183 catalog_manager.cc:6719] ReplaceTablet: tablet 9820e68ad5264df1a79b22f3b5369c5a deleted and replaced by tablet e5f58c190a9a4c5391f1330f79865589
I20251024 19:47:57.754773 1884 raft_consensus.cc:515] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.756114 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "922f975a672746b3b72acf4d3050f688" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:47:57.758101 1586 leader_election.cc:343] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:47:57.758571 1305 leader_election.cc:343] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.758737 1305 leader_election.cc:304] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:57.759104 1884 leader_election.cc:290] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:47:57.759563 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f346a90a9efe442f82750336e3893deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.759723 1774 ts_tablet_manager.cc:1916] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.764205 1865 raft_consensus.cc:2749] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:57.766659 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet b9b1e0338fd14634a1e921735ae61289 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:47:57.766851 1308 leader_election.cc:343] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.767120 1782 raft_consensus.cc:697] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:47:57.763469 1340 tablet_service.cc:1505] Processing CreateTablet for tablet a48249f1824a40b087b80a563816d485 (DEFAULT_TABLE table=default.Table_f96cb6877d844a50aacb91514a539fee [id=d9ab6489833b4989a69c975d734bcd2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.768857 1305 leader_election.cc:304] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
W20251024 19:47:57.769423 1305 leader_election.cc:343] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:57.769904 1305 leader_election.cc:304] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:57.770336 1865 raft_consensus.cc:2804] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:47:57.770941 1865 raft_consensus.cc:493] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:47:57.771070 1865 raft_consensus.cc:3060] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:57.771091 1766 tablet_bootstrap.cc:492] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:57.771282 1766 ts_tablet_manager.cc:1403] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.361s user 0.019s sys 0.028s
I20251024 19:47:57.772305 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "922f975a672746b3b72acf4d3050f688" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:47:57.772325 1766 raft_consensus.cc:359] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:57.772508 1766 raft_consensus.cc:385] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.772612 1766 raft_consensus.cc:740] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.773237 1782 consensus_queue.cc:237] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:47:57.773425 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5473699c1b647f8b86ee43fdb56bdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.776849 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet b9b1e0338fd14634a1e921735ae61289 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.777148 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48249f1824a40b087b80a563816d485. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.780277 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 31647a48fb834c80b6920c756a94250d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.720095 1482 tablet_service.cc:1505] Processing CreateTablet for tablet e3b4e85f741849b2a6a3e8574396ff88 (DEFAULT_TABLE table=default.Table_8c5e539b72dd4150adca95d805acce44 [id=1da47e6010f548d1b3d8c538150a8a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.782292 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b4e85f741849b2a6a3e8574396ff88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.785710 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 65f646bd05274cb6859c0b7f92b61352 with delete_type TABLET_DATA_DELETED (replaced by tablet aebf76d4a66947ed9756643740418798) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.790269 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a1f1892bd3b4776adc7c2519fcdf411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.790938 1775 ts_tablet_manager.cc:1916] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.795899 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 608696ec90ed45e1ad939ed2bfc88d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.796892 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 230606a0ce084b7bab0e41e48cf8fa89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.792268 1865 raft_consensus.cc:515] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.795078 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 608696ec90ed45e1ad939ed2bfc88d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.795925 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a1f1892bd3b4776adc7c2519fcdf411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.798447 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 9820e68ad5264df1a79b22f3b5369c5a with delete_type TABLET_DATA_DELETED (replaced by tablet e5f58c190a9a4c5391f1330f79865589) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.798810 1766 consensus_queue.cc:260] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:57.799153 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 230606a0ce084b7bab0e41e48cf8fa89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.799536 1766 ts_tablet_manager.cc:1434] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20251024 19:47:57.800395 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 9820e68ad5264df1a79b22f3b5369c5a with delete_type TABLET_DATA_DELETED (replaced by tablet e5f58c190a9a4c5391f1330f79865589) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.801342 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet fb17a35d8d8446cf9870d05e107ed237 with delete_type TABLET_DATA_DELETED (replaced by tablet ca0527af718f494a8bbcc66e7b325bed) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.806627 1371 raft_consensus.cc:2393] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 43bec2f03fc849099e254f6335e9d532 in this term.
I20251024 19:47:57.806751 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 3caecdff61ce48f7b7c3846bb380d21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.807668 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 9820e68ad5264df1a79b22f3b5369c5a with delete_type TABLET_DATA_DELETED (replaced by tablet e5f58c190a9a4c5391f1330f79865589) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.807924 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet fb17a35d8d8446cf9870d05e107ed237 with delete_type TABLET_DATA_DELETED (replaced by tablet ca0527af718f494a8bbcc66e7b325bed) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.810684 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet 65f646bd05274cb6859c0b7f92b61352 with delete_type TABLET_DATA_DELETED (replaced by tablet aebf76d4a66947ed9756643740418798) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.820137 1586 leader_election.cc:304] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:47:57.820746 1884 raft_consensus.cc:3060] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:57.822847 1882 raft_consensus.cc:2749] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:57.825388 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 3caecdff61ce48f7b7c3846bb380d21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.825542 1884 raft_consensus.cc:2749] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:57.826730 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet fb17a35d8d8446cf9870d05e107ed237 with delete_type TABLET_DATA_DELETED (replaced by tablet ca0527af718f494a8bbcc66e7b325bed) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:57.833153 1619 tablet_service.cc:1552] Processing DeleteTablet for tablet 3caecdff61ce48f7b7c3846bb380d21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:57.833783 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "922f975a672746b3b72acf4d3050f688" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:47:57.839689 1648 raft_consensus.cc:2468] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:47:57.840468 1305 leader_election.cc:304] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:47:57.842978 1774 ts_tablet_manager.cc:1929] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.842069 1865 leader_election.cc:290] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:57.849735 1882 raft_consensus.cc:2804] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:47:57.850692 1776 ts_tablet_manager.cc:1929] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.851920 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "922f975a672746b3b72acf4d3050f688" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:47:57.853116 1766 tablet_bootstrap.cc:492] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:57.853289 1774 log.cc:1199] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/07d66fe64bff4a00a15554f212456698
W20251024 19:47:57.853617 1308 leader_election.cc:343] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: Tablet not RUNNING: STOPPED
I20251024 19:47:57.856710 1776 log.cc:1199] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/889bd73511b945b4a4e751f60929a5dc
I20251024 19:47:57.856596 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4c8922124e774c8aa7e7a2db295099c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.857520 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c8922124e774c8aa7e7a2db295099c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.857929 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8456d5c39d4b4ac985982e8eadfa2750"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.858500 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8456d5c39d4b4ac985982e8eadfa2750 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.862296 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a81e7aae0a7a4f3db5c84efc4b16f873"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.862954 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a81e7aae0a7a4f3db5c84efc4b16f873 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.863576 1776 ts_tablet_manager.cc:1950] T 889bd73511b945b4a4e751f60929a5dc P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:57.855058 1612 tablet_service.cc:1505] Processing CreateTablet for tablet e3b4e85f741849b2a6a3e8574396ff88 (DEFAULT_TABLE table=default.Table_8c5e539b72dd4150adca95d805acce44 [id=1da47e6010f548d1b3d8c538150a8a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.864586 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b4e85f741849b2a6a3e8574396ff88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.866638 1774 ts_tablet_manager.cc:1950] T 07d66fe64bff4a00a15554f212456698 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:57.868538 1774 tablet_replica.cc:333] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:57.868605 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 81f8393f78204860bd854f4872099387 (DEFAULT_TABLE table=default.Table_2d74dcbcd9f249e5b10e1986cc0c0d18 [id=0bd294ee0c8341a6a29ce26caeeceb22]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.869299 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f8393f78204860bd854f4872099387. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.880906 1882 raft_consensus.cc:697] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:47:57.882754 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d5728217650a46e482b97caa10fea9a2" } new_table_name: "default.Table_e0d4700f7fd849399f95eb388d194091" modify_external_catalogs: true
I20251024 19:47:57.883270 1882 consensus_queue.cc:237] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:57.887522 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9edfe580f57e43919a206badbc0bbfd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:57.887957 1774 raft_consensus.cc:2243] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:47:57.888487 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9edfe580f57e43919a206badbc0bbfd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.888657 1774 raft_consensus.cc:2272] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.889400 1774 ts_tablet_manager.cc:1916] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:47:57.894945 1825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 19:47:57.895418 1825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 19:47:57.895606 1825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251024 19:47:57.898471 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7c34d3a8affd4e69a31b276ccbe096ae" } new_table_name: "default.Table_48177e84ca024d49a2fd4e205a08a2a9" modify_external_catalogs: true
I20251024 19:47:57.902078 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ee4ef17d5f314379846d30a477d4911d" } new_table_name: "default.Table_cdf2e8872b724c7aa86d5091eb52fb1c" modify_external_catalogs: true
I20251024 19:47:57.902611 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_59fbd07181e04f8495abd07bda1271c0" } new_table_name: "default.Table_48b3dfe737b54b9a8618c0a658e87e16" modify_external_catalogs: true
I20251024 19:47:57.902971 1189 master_service.cc:954] ReplaceTablet: received request to replace tablet 2e956ea914c545d79be6d632bfc4ec57 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:57.902628 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1142a96660b74e09be367f32b979af21" } new_table_name: "default.Table_5f1d0b12b2034f5185f21c87157cf051" modify_external_catalogs: true
I20251024 19:47:57.904663 1185 catalog_manager.cc:5649] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } }
I20251024 19:47:57.905640 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet f1ad65fd080e452e8094975d420336c7 (table default.Table_a54ffafa18b848b79a290f58c9754bec [id=2fd5c541b06a46469e65ed0942d2c394]). Expected version 4 got 0
I20251024 19:47:57.911031 1339 tablet_service.cc:1505] Processing CreateTablet for tablet e3b4e85f741849b2a6a3e8574396ff88 (DEFAULT_TABLE table=default.Table_8c5e539b72dd4150adca95d805acce44 [id=1da47e6010f548d1b3d8c538150a8a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.912109 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b4e85f741849b2a6a3e8574396ff88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.914590 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 81f8393f78204860bd854f4872099387 (DEFAULT_TABLE table=default.Table_2d74dcbcd9f249e5b10e1986cc0c0d18 [id=0bd294ee0c8341a6a29ce26caeeceb22]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.916335 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 889bd73511b945b4a4e751f60929a5dc (table default.Table_a2fc8ac42bbc44af8bf194f2d3511bd5 [id=5518afdf09da4433bee8f2221ead84c1]) successfully deleted
W20251024 19:47:57.916687 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c already in progress: creating tablet
W20251024 19:47:57.916950 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet cc0bf7aee1a1421b9d985020a1674639 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc0bf7aee1a1421b9d985020a1674639 already in progress: creating tablet
W20251024 19:47:57.917212 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 31647a48fb834c80b6920c756a94250d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31647a48fb834c80b6920c756a94250d already in progress: creating tablet
I20251024 19:47:57.917650 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f8393f78204860bd854f4872099387. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:57.921196 1825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 19:47:57.921396 1825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 19:47:57.921553 1825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 19:47:57.921702 1825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 19:47:57.926380 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 44ee03ede02c4bdab706fde910978332 (DEFAULT_TABLE table=default.Table_1142a96660b74e09be367f32b979af21 [id=46806f1d4f434926bc23214f0e3605d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.929483 1766 tablet_bootstrap.cc:654] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.932519 1767 ts_tablet_manager.cc:1434] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.185s user 0.015s sys 0.073s
I20251024 19:47:57.933693 1768 tablet_bootstrap.cc:492] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:57.933692 1619 tablet_service.cc:1505] Processing CreateTablet for tablet dc1b6f0a377945089c50012f74d38564 (DEFAULT_TABLE table=default.Table_52ec55c23c394294a68850ab6b33aabe [id=fb0e7cab20f44aaebdb09862bd193f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.934161 1768 ts_tablet_manager.cc:1403] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.250s user 0.006s sys 0.000s
I20251024 19:47:57.934701 1776 tablet_replica.cc:333] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:57.934957 1776 raft_consensus.cc:2243] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:57.935035 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1b6f0a377945089c50012f74d38564. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.935184 1776 raft_consensus.cc:2272] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:57.935788 1776 ts_tablet_manager.cc:1916] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:57.936053 1768 raft_consensus.cc:359] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:57.936399 1768 raft_consensus.cc:385] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:57.936540 1768 raft_consensus.cc:740] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:57.937994 1767 tablet_bootstrap.cc:492] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:57.937139 1768 consensus_queue.cc:260] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:57.940241 1825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.252.212:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.252.254:36505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.252.254
--webserver_port=42051
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 24 Oct 2025 19:43:25 UTC on 24a791456cd2
build id 8700
ASAN enabled
I20251024 19:47:57.941725 1767 tablet_bootstrap.cc:654] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.942764 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ee03ede02c4bdab706fde910978332. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.946408 1825 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 19:47:57.946820 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 07d66fe64bff4a00a15554f212456698 (table default.Table_7c34d3a8affd4e69a31b276ccbe096ae [id=aadca5a1b6d4494aa37f8a8cc76ad12a]) successfully deleted
I20251024 19:47:57.946977 1768 ts_tablet_manager.cc:1434] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20251024 19:47:57.949095 1825 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251024 19:47:57.953683 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 1d04a9b54bd6446d8deb64292cef672d (DEFAULT_TABLE table=default.Table_2cbb0aad326e4bae85d4c5cb833150c6 [id=1acdc5d0bfb94e0faf47055b20d803b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.954267 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d04a9b54bd6446d8deb64292cef672d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.957485 1768 tablet_bootstrap.cc:492] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:57.957846 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 030e690ce4df4818a6450ae293b0ffbd (DEFAULT_TABLE table=default.Table_4db7a2da5af4481db5b546d201dfd278 [id=2ac38aaa9204452787b18173393bb949]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.958253 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030e690ce4df4818a6450ae293b0ffbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.961925 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 8a1f1892bd3b4776adc7c2519fcdf411 (DEFAULT_TABLE table=default.Table_46b7ffb5bf6f45f2be4d1b242f43bfc9 [id=0cd6320a10064e9a8556836dfdb2a1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.962314 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a1f1892bd3b4776adc7c2519fcdf411. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.962402 1774 ts_tablet_manager.cc:1929] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.963259 1775 ts_tablet_manager.cc:1929] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:57.963397 1775 log.cc:1199] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/922f975a672746b3b72acf4d3050f688
I20251024 19:47:57.963802 1775 ts_tablet_manager.cc:1950] T 922f975a672746b3b72acf4d3050f688 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:57.963933 1774 log.cc:1199] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/1745bbc68d6846fab1a6197f898754b4
I20251024 19:47:57.969072 1774 ts_tablet_manager.cc:1950] T 1745bbc68d6846fab1a6197f898754b4 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:47:57.970134 1895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:47:57.977200 1896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 19:47:57.980350 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 81f8393f78204860bd854f4872099387 (DEFAULT_TABLE table=default.Table_2d74dcbcd9f249e5b10e1986cc0c0d18 [id=0bd294ee0c8341a6a29ce26caeeceb22]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.981424 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 44ee03ede02c4bdab706fde910978332 (DEFAULT_TABLE table=default.Table_1142a96660b74e09be367f32b979af21 [id=46806f1d4f434926bc23214f0e3605d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.981817 1338 tablet_service.cc:1505] Processing CreateTablet for tablet dc1b6f0a377945089c50012f74d38564 (DEFAULT_TABLE table=default.Table_52ec55c23c394294a68850ab6b33aabe [id=fb0e7cab20f44aaebdb09862bd193f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.982301 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f8393f78204860bd854f4872099387. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.985353 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1b6f0a377945089c50012f74d38564. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.986131 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ee03ede02c4bdab706fde910978332. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.988111 1768 tablet_bootstrap.cc:654] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:57.988485 1618 tablet_service.cc:1505] Processing CreateTablet for tablet fd52295b14b14e70bbdcb45f1bd849e4 (DEFAULT_TABLE table=default.Table_ff7a43292b9045f9bbfa71883bcd7e61 [id=e2c539db2aa148448925a69f4848feb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.989885 1825 server_base.cc:1047] running on GCE node
I20251024 19:47:57.992684 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet fc8c01a4c07a42a0bc48a58313090e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:57.993033 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 1d04a9b54bd6446d8deb64292cef672d (DEFAULT_TABLE table=default.Table_2cbb0aad326e4bae85d4c5cb833150c6 [id=1acdc5d0bfb94e0faf47055b20d803b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.993162 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd52295b14b14e70bbdcb45f1bd849e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.993553 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d04a9b54bd6446d8deb64292cef672d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.980646 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 44ee03ede02c4bdab706fde910978332 (DEFAULT_TABLE table=default.Table_1142a96660b74e09be367f32b979af21 [id=46806f1d4f434926bc23214f0e3605d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.994632 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ee03ede02c4bdab706fde910978332. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:57.996848 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_18392df3579642bf80ac127778fd508f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:57.997475 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18392df3579642bf80ac127778fd508f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:57.997822 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 030e690ce4df4818a6450ae293b0ffbd (DEFAULT_TABLE table=default.Table_4db7a2da5af4481db5b546d201dfd278 [id=2ac38aaa9204452787b18173393bb949]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:57.998186 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030e690ce4df4818a6450ae293b0ffbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.000703 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 8a1f1892bd3b4776adc7c2519fcdf411 (DEFAULT_TABLE table=default.Table_46b7ffb5bf6f45f2be4d1b242f43bfc9 [id=0cd6320a10064e9a8556836dfdb2a1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.001044 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a1f1892bd3b4776adc7c2519fcdf411. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:58.001263 1900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 19:47:58.002962 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cabda156901b4ee38d0be4dd4efe8376" } modify_external_catalogs: true
I20251024 19:47:58.003175 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cabda156901b4ee38d0be4dd4efe8376" } modify_external_catalogs: true
I20251024 19:47:58.003540 1484 tablet_service.cc:1505] Processing CreateTablet for tablet fd52295b14b14e70bbdcb45f1bd849e4 (DEFAULT_TABLE table=default.Table_ff7a43292b9045f9bbfa71883bcd7e61 [id=e2c539db2aa148448925a69f4848feb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.003888 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd52295b14b14e70bbdcb45f1bd849e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.004644 1825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20251024 19:47:58.004773 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 979dd87a8c66480ea314fcc75fa878c3 consensus state has no leader
I20251024 19:47:58.005548 1489 tablet_service.cc:1505] Processing CreateTablet for tablet dc1b6f0a377945089c50012f74d38564 (DEFAULT_TABLE table=default.Table_52ec55c23c394294a68850ab6b33aabe [id=fb0e7cab20f44aaebdb09862bd193f41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.006059 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1b6f0a377945089c50012f74d38564. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.017170 1766 tablet_bootstrap.cc:492] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
W20251024 19:47:58.019753 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2346c03802a245a2af8c086b626d660d consensus state has no leader
W20251024 19:47:58.020190 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aebf76d4a66947ed9756643740418798 has no consensus state
W20251024 19:47:58.020560 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f91316c030dc4f7da27977a9dde01f4b consensus state has no leader
W20251024 19:47:58.020949 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44ee03ede02c4bdab706fde910978332 consensus state has no leader
I20251024 19:47:58.022641 1766 ts_tablet_manager.cc:1403] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.170s user 0.003s sys 0.003s
I20251024 19:47:58.022972 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_74349eed6d4b4e8e95ae01909bbb91e9" } modify_external_catalogs: true
I20251024 19:47:58.023134 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_74349eed6d4b4e8e95ae01909bbb91e9" } modify_external_catalogs: true
I20251024 19:47:58.023573 1766 raft_consensus.cc:359] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:58.023802 1766 raft_consensus.cc:385] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.023878 1766 raft_consensus.cc:740] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.024114 1766 consensus_queue.cc:260] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:58.025743 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_61b7f6c428aa4bd9a65c1d66171c59c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.026307 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61b7f6c428aa4bd9a65c1d66171c59c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.028281 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0680f9139a8a436a80fce09c095912ce" } modify_external_catalogs: true
I20251024 19:47:58.028462 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0680f9139a8a436a80fce09c095912ce" } modify_external_catalogs: true
I20251024 19:47:58.030493 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_34bbddd692a44f46a4925c957f4e8848"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.030989 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34bbddd692a44f46a4925c957f4e8848 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.031761 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_673f2f271b3e4f9d9c5f6802a3646f8f" } modify_external_catalogs: true
I20251024 19:47:58.031939 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_673f2f271b3e4f9d9c5f6802a3646f8f" } modify_external_catalogs: true
I20251024 19:47:58.032244 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_dc366a1f70d84d4f8276c1872b9fa0f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.032739 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc366a1f70d84d4f8276c1872b9fa0f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.034839 1767 tablet_bootstrap.cc:492] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:58.035265 1767 ts_tablet_manager.cc:1403] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.004s
I20251024 19:47:58.036796 1767 raft_consensus.cc:359] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:58.037159 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2e956ea914c545d79be6d632bfc4ec57
I20251024 19:47:58.037184 1767 raft_consensus.cc:385] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.037286 1767 raft_consensus.cc:740] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.037832 1189 catalog_manager.cc:6719] ReplaceTablet: tablet 2e956ea914c545d79be6d632bfc4ec57 deleted and replaced by tablet a3259823b622461fa823841925b3d203
I20251024 19:47:58.038684 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e956ea914c545d79be6d632bfc4ec57 with delete_type TABLET_DATA_DELETED (replaced by tablet a3259823b622461fa823841925b3d203) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:58.039207 1766 ts_tablet_manager.cc:1434] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251024 19:47:58.039737 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e956ea914c545d79be6d632bfc4ec57 with delete_type TABLET_DATA_DELETED (replaced by tablet a3259823b622461fa823841925b3d203) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.040046 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d941eafb1414f068dc656946082f617 with delete_type TABLET_DATA_DELETED (replaced by tablet f91316c030dc4f7da27977a9dde01f4b) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.040459 1767 consensus_queue.cc:260] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:58.043756 1766 tablet_bootstrap.cc:492] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:58.045523 1766 tablet_bootstrap.cc:654] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.053203 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ffaa4997edd4b1187579958037ea793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.053916 1767 ts_tablet_manager.cc:1434] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20251024 19:47:58.054296 1191 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet d64b4e5a8fe349ccb053c6a08315c347
I20251024 19:47:58.055001 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 8a1f1892bd3b4776adc7c2519fcdf411 (DEFAULT_TABLE table=default.Table_46b7ffb5bf6f45f2be4d1b242f43bfc9 [id=0cd6320a10064e9a8556836dfdb2a1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.055619 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 1d04a9b54bd6446d8deb64292cef672d (DEFAULT_TABLE table=default.Table_2cbb0aad326e4bae85d4c5cb833150c6 [id=1acdc5d0bfb94e0faf47055b20d803b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.055970 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c700d3a5bb0c4963ace8624ae2ef882d" } modify_external_catalogs: true
I20251024 19:47:58.056228 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c700d3a5bb0c4963ace8624ae2ef882d" } modify_external_catalogs: true
I20251024 19:47:58.056394 1341 tablet_service.cc:1505] Processing CreateTablet for tablet fd52295b14b14e70bbdcb45f1bd849e4 (DEFAULT_TABLE table=default.Table_ff7a43292b9045f9bbfa71883bcd7e61 [id=e2c539db2aa148448925a69f4848feb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.056567 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 030e690ce4df4818a6450ae293b0ffbd (DEFAULT_TABLE table=default.Table_4db7a2da5af4481db5b546d201dfd278 [id=2ac38aaa9204452787b18173393bb949]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.060495 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_dd9b9ef96622439ea016c3813fcbb81a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.061108 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd9b9ef96622439ea016c3813fcbb81a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.062259 1767 tablet_bootstrap.cc:492] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:58.064548 1825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 19:47:58.065825 1825 hybrid_clock.cc:648] HybridClock initialized: now 1761335278065729 us; error 86 us; skew 500 ppm
I20251024 19:47:58.065891 1767 tablet_bootstrap.cc:654] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.066480 1825 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251024 19:47:58.067600 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e956ea914c545d79be6d632bfc4ec57 with delete_type TABLET_DATA_DELETED (replaced by tablet a3259823b622461fa823841925b3d203) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:58.069897 1776 ts_tablet_manager.cc:1929] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:58.070314 1776 log.cc:1199] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/922f975a672746b3b72acf4d3050f688
I20251024 19:47:58.071012 1776 ts_tablet_manager.cc:1950] T 922f975a672746b3b72acf4d3050f688 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
W20251024 19:47:58.071986 1177 consensus_peers.cc:597] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c -> Peer ef8906492b1a4368936fac95c11e1f73 (127.0.252.254:36505): Couldn't send request to peer ef8906492b1a4368936fac95c11e1f73. Status: Network error: Client connection negotiation failed: client connection to 127.0.252.254:36505: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251024 19:47:58.074373 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 922f975a672746b3b72acf4d3050f688 (table default.Table_ab42d318a6cc4626be3d62dda5948bf0 [id=eab377d5ce1343d7a12c3740fe46beb5]) successfully deleted
W20251024 19:47:58.074638 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b9b1e0338fd14634a1e921735ae61289 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9b1e0338fd14634a1e921735ae61289 already in progress: creating tablet
W20251024 19:47:58.074841 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 608696ec90ed45e1ad939ed2bfc88d34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 608696ec90ed45e1ad939ed2bfc88d34 already in progress: creating tablet
W20251024 19:47:58.075035 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 65f646bd05274cb6859c0b7f92b61352 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65f646bd05274cb6859c0b7f92b61352 already in progress: creating tablet
W20251024 19:47:58.075240 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 8a1f1892bd3b4776adc7c2519fcdf411 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a1f1892bd3b4776adc7c2519fcdf411 already in progress: creating tablet
W20251024 19:47:58.075470 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 230606a0ce084b7bab0e41e48cf8fa89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 230606a0ce084b7bab0e41e48cf8fa89 already in progress: creating tablet
W20251024 19:47:58.075699 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9820e68ad5264df1a79b22f3b5369c5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9820e68ad5264df1a79b22f3b5369c5a already in progress: creating tablet
W20251024 19:47:58.076273 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet fb17a35d8d8446cf9870d05e107ed237 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb17a35d8d8446cf9870d05e107ed237 already in progress: creating tablet
I20251024 19:47:58.076833 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7dbe5740202642f1be406c4897a72541"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.077752 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dbe5740202642f1be406c4897a72541 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.078172 1776 tablet_replica.cc:333] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:58.079939 1776 raft_consensus.cc:2243] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:58.080370 1776 raft_consensus.cc:2272] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:58.079545 1767 tablet_bootstrap.cc:492] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:58.080750 1767 ts_tablet_manager.cc:1403] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
W20251024 19:47:58.081408 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3caecdff61ce48f7b7c3846bb380d21e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3caecdff61ce48f7b7c3846bb380d21e already in progress: creating tablet
W20251024 19:47:58.081765 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2e956ea914c545d79be6d632bfc4ec57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e956ea914c545d79be6d632bfc4ec57 already in progress: creating tablet
I20251024 19:47:58.082082 1767 raft_consensus.cc:359] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:58.082388 1767 raft_consensus.cc:385] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.082567 1767 raft_consensus.cc:740] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.082966 1767 consensus_queue.cc:260] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:58.084896 1767 ts_tablet_manager.cc:1434] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251024 19:47:58.086639 1767 tablet_bootstrap.cc:492] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:58.090178 1767 tablet_bootstrap.cc:654] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.090324 1776 ts_tablet_manager.cc:1916] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:58.091830 1192 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet e5f58c190a9a4c5391f1330f79865589
I20251024 19:47:58.092167 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f346a90a9efe442f82750336e3893deb
I20251024 19:47:58.094954 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b2ea545b2cc14f589eda9419e55234bf
I20251024 19:47:58.100418 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ea545b2cc14f589eda9419e55234bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:58.100840 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet f346a90a9efe442f82750336e3893deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:58.102852 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet f346a90a9efe442f82750336e3893deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.103145 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ea545b2cc14f589eda9419e55234bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.103370 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 608696ec90ed45e1ad939ed2bfc88d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.105823 1767 tablet_bootstrap.cc:492] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:58.106134 1767 ts_tablet_manager.cc:1403] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.002s
I20251024 19:47:58.106310 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_873d6a3a39344554b0ca73c575129c36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.107331 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_873d6a3a39344554b0ca73c575129c36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.107440 1767 raft_consensus.cc:359] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.107745 1767 raft_consensus.cc:385] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.107815 1767 raft_consensus.cc:740] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.108139 1767 consensus_queue.cc:260] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.107758 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_64aaab6eda0a443fbb49223835af43e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:58.109856 1767 ts_tablet_manager.cc:1434] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
W20251024 19:47:58.109917 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64aaab6eda0a443fbb49223835af43e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.110880 1776 ts_tablet_manager.cc:1929] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:58.111029 1776 log.cc:1199] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/2d941eafb1414f068dc656946082f617
I20251024 19:47:58.111482 1776 ts_tablet_manager.cc:1950] T 2d941eafb1414f068dc656946082f617 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:58.108297 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_59612aa90b7540f6bf9cfc72e1f68fc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.112640 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59612aa90b7540f6bf9cfc72e1f68fc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.112661 1767 tablet_bootstrap.cc:492] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:58.113960 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 2d941eafb1414f068dc656946082f617 (table default.Table_cdf2e8872b724c7aa86d5091eb52fb1c [id=a402387542654ccf8e0b04ca8ecebb1a]) successfully deleted
I20251024 19:47:58.116231 1767 tablet_bootstrap.cc:654] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.117522 1776 tablet_replica.cc:333] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:58.117898 1776 raft_consensus.cc:2243] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:58.117995 1158 master_runner.cc:186] Error getting master registration for 127.0.252.254:36505: Network error: Client connection negotiation failed: client connection to 127.0.252.254:36505: connect: Connection refused (error 111),
I20251024 19:47:58.118170 1158 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251024 19:47:58.118172 1776 raft_consensus.cc:2272] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:58.119292 1776 ts_tablet_manager.cc:1916] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:58.119939 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet fc8c01a4c07a42a0bc48a58313090e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.129022 1776 ts_tablet_manager.cc:1929] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:58.129130 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet f346a90a9efe442f82750336e3893deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:58.130515 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 922f975a672746b3b72acf4d3050f688 (table default.Table_ab42d318a6cc4626be3d62dda5948bf0 [id=eab377d5ce1343d7a12c3740fe46beb5]) successfully deleted
W20251024 19:47:58.131129 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b9b1e0338fd14634a1e921735ae61289 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9b1e0338fd14634a1e921735ae61289 already in progress: creating tablet
W20251024 19:47:58.131366 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 65f646bd05274cb6859c0b7f92b61352 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65f646bd05274cb6859c0b7f92b61352 already in progress: creating tablet
W20251024 19:47:58.131632 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 8a1f1892bd3b4776adc7c2519fcdf411 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a1f1892bd3b4776adc7c2519fcdf411 already in progress: creating tablet
W20251024 19:47:58.131852 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 608696ec90ed45e1ad939ed2bfc88d34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 608696ec90ed45e1ad939ed2bfc88d34 already in progress: creating tablet
W20251024 19:47:58.132100 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 230606a0ce084b7bab0e41e48cf8fa89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 230606a0ce084b7bab0e41e48cf8fa89 already in progress: creating tablet
I20251024 19:47:58.132398 1775 tablet_replica.cc:333] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:58.132576 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ea545b2cc14f589eda9419e55234bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:58.132854 1775 raft_consensus.cc:2243] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:58.133147 1775 raft_consensus.cc:2272] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:58.136152 1776 log.cc:1199] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/0ffaa4997edd4b1187579958037ea793
I20251024 19:47:58.136406 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d04a9b54bd6446d8deb64292cef672d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.136983 1776 ts_tablet_manager.cc:1950] T 0ffaa4997edd4b1187579958037ea793 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:58.140935 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b5473699c1b647f8b86ee43fdb56bdfd
I20251024 19:47:58.141767 1776 tablet_replica.cc:333] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:58.142061 1776 raft_consensus.cc:2243] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:58.142663 1776 raft_consensus.cc:2272] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:58.143647 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 0ffaa4997edd4b1187579958037ea793 (table default.Table_cf69470650f643c195eb5fbecbd2d2c4 [id=c1c8743e95d34655b1a8779920ed837a]) successfully deleted
I20251024 19:47:58.143848 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet b5473699c1b647f8b86ee43fdb56bdfd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:47:58.143914 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f346a90a9efe442f82750336e3893deb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f346a90a9efe442f82750336e3893deb already in progress: creating tablet
W20251024 19:47:58.144171 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b2ea545b2cc14f589eda9419e55234bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2ea545b2cc14f589eda9419e55234bf already in progress: creating tablet
I20251024 19:47:58.144586 1776 ts_tablet_manager.cc:1916] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:58.145552 1775 ts_tablet_manager.cc:1916] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:58.145641 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd52295b14b14e70bbdcb45f1bd849e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.162631 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 5368789df6024d7daa9ccc2257137d58 (DEFAULT_TABLE table=default.Table_a81e7aae0a7a4f3db5c84efc4b16f873 [id=16bcec63f27d4dcd826ef11d45d58400]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.162919 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a1f1892bd3b4776adc7c2519fcdf411. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.163254 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5368789df6024d7daa9ccc2257137d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.167870 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 037c8cc1e53447109fafc047c02f196f (DEFAULT_TABLE table=default.Table_4c8922124e774c8aa7e7a2db295099c3 [id=5bc052b3e0f04699b4b391bcec33617d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.168157 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030e690ce4df4818a6450ae293b0ffbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.168283 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 037c8cc1e53447109fafc047c02f196f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.169867 1618 tablet_service.cc:1505] Processing CreateTablet for tablet a76d61c3464b41519d7657796bd853ff (DEFAULT_TABLE table=default.Table_8456d5c39d4b4ac985982e8eadfa2750 [id=ae6b320a09934ac5873d3b35c33f2c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.170251 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a76d61c3464b41519d7657796bd853ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.171949 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet b5473699c1b647f8b86ee43fdb56bdfd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:58.174268 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 4db17d9aff6542b6b94c09f872a4cceb (DEFAULT_TABLE table=default.Table_e9bc5ade2870426abe90a5f3b20ebb5e [id=e4ee1d33b7e24827aa24950ea40465e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.174736 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4db17d9aff6542b6b94c09f872a4cceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.178191 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet b5473699c1b647f8b86ee43fdb56bdfd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:58.178694 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 452f77091adc4cf09de2231f2f6d845b (DEFAULT_TABLE table=default.Table_573ec53c3d2c4de8864f399b291d8aa9 [id=9a49120c5b654676a63cd3966eaf31a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.179185 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452f77091adc4cf09de2231f2f6d845b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.179164 1775 ts_tablet_manager.cc:1929] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:58.179304 1775 log.cc:1199] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/fb17a35d8d8446cf9870d05e107ed237
I20251024 19:47:58.179733 1775 ts_tablet_manager.cc:1950] T fb17a35d8d8446cf9870d05e107ed237 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:58.181067 1775 tablet_replica.cc:333] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:58.181231 1775 raft_consensus.cc:2243] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:58.181429 1775 raft_consensus.cc:2272] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:58.181902 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 3754540ae21e4c65b5af3d0aa3f9189f (DEFAULT_TABLE table=default.Table_eb1e63c7ef2244259c8a64fd4fe3859b [id=fec3ddedec0b44db80b30c899043bc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.182323 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3754540ae21e4c65b5af3d0aa3f9189f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.186105 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 0a12e415590a4c71aab4ad7d30412c83 (DEFAULT_TABLE table=default.Table_5a1d470fc7bc493393e47da7db62dfca [id=18ec057e920e4c21ae8cbe298058b6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.186645 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a12e415590a4c71aab4ad7d30412c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.188663 1615 tablet_service.cc:1505] Processing CreateTablet for tablet aebf76d4a66947ed9756643740418798 (DEFAULT_TABLE table=default.Table_48b3dfe737b54b9a8618c0a658e87e16 [id=665389c7bde54b7786edabcb91a9b33b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.189126 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aebf76d4a66947ed9756643740418798. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.191813 1825 webserver.cc:492] Webserver started at http://127.0.252.254:42051/ using document root <none> and password file <none>
I20251024 19:47:58.193225 1825 fs_manager.cc:362] Metadata directory not provided
I20251024 19:47:58.193428 1825 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 19:47:58.193719 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 85) took 1277 ms. Trace:
I20251024 19:47:58.193867 1774 rpcz_store.cc:276] 1024 19:47:56.899606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:56.899667 (+ 61us) service_pool.cc:225] Handling call
1024 19:47:57.868430 (+968763us) ts_tablet_manager.cc:1151] Deleting tablet 1745bbc68d6846fab1a6197f898754b4
1024 19:47:57.868444 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:57.900101 (+ 31657us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:57.961535 (+ 61434us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:58.169545 (+208010us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x61800000d518. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f9883bec2fd 00007f9883bed36e 00007f989781fc9d 00007f989779e203 00007f98939d7630 00007f98939e172a 00007f9893a0168a 00007f9893a01660 00007f9893a0162e 00007f98939ee317
1024 19:47:58.177186 (+ 7641us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39957376,"tablet-delete.queue_time_us":968323}
I20251024 19:47:58.194586 1774 tablet_replica.cc:333] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:58.194567 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 5fd7726c532145c6b44ea9a37544510d (DEFAULT_TABLE table=default.Table_999546c6f492460bb6dc96ab55b7d3ad [id=7dbbe53e70a547888a510250e0583e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.194841 1774 raft_consensus.cc:2243] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:58.194922 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fd7726c532145c6b44ea9a37544510d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.194886 1617 tablet_service.cc:1505] Processing CreateTablet for tablet ad055dc3387b414bb55d09b9683ba17e (DEFAULT_TABLE table=default.Table_97c6fc690615425aa5af0d8c0dd249af [id=c3de6937e93a48dcb88cabee59a09d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.195065 1774 raft_consensus.cc:2272] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:58.195269 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad055dc3387b414bb55d09b9683ba17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.195654 1774 ts_tablet_manager.cc:1916] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:58.197857 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 6ec5c7bf36874423b4790c6169da8bd0 (DEFAULT_TABLE table=default.Table_52ecaf891a2449d78bb8a354e7fb904b [id=e0a69533d0fa41a4ad58f08e510fa39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.198166 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec5c7bf36874423b4790c6169da8bd0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:58.200783 1177 catalog_manager.cc:5101] could not convert tablet's f1ad65fd080e452e8094975d420336c7 schema from PB: OK
I20251024 19:47:58.201093 1623 tablet_service.cc:1505] Processing CreateTablet for tablet ce0ef4443e574846a9f48fef3b893352 (DEFAULT_TABLE table=default.Table_ce25ea67811e4818be50de93930ab68e [id=ec57efb4d37e44f1b68045f58bc11c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.201555 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce0ef4443e574846a9f48fef3b893352. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.204254 1624 tablet_service.cc:1505] Processing CreateTablet for tablet ca0527af718f494a8bbcc66e7b325bed (DEFAULT_TABLE table=default.Table_51982d999b0847ec87fc62a473782a5c [id=9a05bcf7ad5d430c9ef6d6e18a27808e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.204689 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca0527af718f494a8bbcc66e7b325bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.206971 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 5368789df6024d7daa9ccc2257137d58 (DEFAULT_TABLE table=default.Table_a81e7aae0a7a4f3db5c84efc4b16f873 [id=16bcec63f27d4dcd826ef11d45d58400]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.207350 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5368789df6024d7daa9ccc2257137d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.208635 1825 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.003s sys 0.007s
I20251024 19:47:58.209988 1621 tablet_service.cc:1505] Processing CreateTablet for tablet d692e1988d13460fb1d54e930e17fc9d (DEFAULT_TABLE table=default.Table_f46ed7e555b3432b95a3d9b829968b8e [id=67a0be12c85648bdb4515a096db21d3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.210422 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d692e1988d13460fb1d54e930e17fc9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.210649 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 037c8cc1e53447109fafc047c02f196f (DEFAULT_TABLE table=default.Table_4c8922124e774c8aa7e7a2db295099c3 [id=5bc052b3e0f04699b4b391bcec33617d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.211166 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 037c8cc1e53447109fafc047c02f196f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.214893 1486 tablet_service.cc:1505] Processing CreateTablet for tablet a76d61c3464b41519d7657796bd853ff (DEFAULT_TABLE table=default.Table_8456d5c39d4b4ac985982e8eadfa2750 [id=ae6b320a09934ac5873d3b35c33f2c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.215293 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a76d61c3464b41519d7657796bd853ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.215440 1338 tablet_service.cc:1505] Processing CreateTablet for tablet a76d61c3464b41519d7657796bd853ff (DEFAULT_TABLE table=default.Table_8456d5c39d4b4ac985982e8eadfa2750 [id=ae6b320a09934ac5873d3b35c33f2c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.216023 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a76d61c3464b41519d7657796bd853ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.218357 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 4db17d9aff6542b6b94c09f872a4cceb (DEFAULT_TABLE table=default.Table_e9bc5ade2870426abe90a5f3b20ebb5e [id=e4ee1d33b7e24827aa24950ea40465e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.218814 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4db17d9aff6542b6b94c09f872a4cceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.219641 1911 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 19:47:58.221594 1825 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.007s sys 0.002s
I20251024 19:47:58.221925 1825 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
uuid: "ef8906492b1a4368936fac95c11e1f73"
format_stamp: "Formatted at 2025-10-24 19:47:51 on dist-test-slave-53sn"
I20251024 19:47:58.223420 1825 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 19:47:58.226215 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 4db17d9aff6542b6b94c09f872a4cceb (DEFAULT_TABLE table=default.Table_e9bc5ade2870426abe90a5f3b20ebb5e [id=e4ee1d33b7e24827aa24950ea40465e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.226657 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4db17d9aff6542b6b94c09f872a4cceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.249934 1629 tablet_service.cc:1505] Processing CreateTablet for tablet b20a6845ae1645c0b51f096d471d4ccc (DEFAULT_TABLE table=default.Table_59612aa90b7540f6bf9cfc72e1f68fc1 [id=201e5232763a4940a1288a56a695f71c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.250546 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b20a6845ae1645c0b51f096d471d4ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.254303 1626 tablet_service.cc:1505] Processing CreateTablet for tablet 9fffc42ddda14bd28800917ff2f0ced5 (DEFAULT_TABLE table=default.Table_64aaab6eda0a443fbb49223835af43e3 [id=52efb5f413d14db28314ecbb058a351f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.254846 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fffc42ddda14bd28800917ff2f0ced5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.258534 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 452f77091adc4cf09de2231f2f6d845b (DEFAULT_TABLE table=default.Table_573ec53c3d2c4de8864f399b291d8aa9 [id=9a49120c5b654676a63cd3966eaf31a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.258749 1627 tablet_service.cc:1505] Processing CreateTablet for tablet b6661b9e9a174849b6093d09cd8c1251 (DEFAULT_TABLE table=default.Table_7dbe5740202642f1be406c4897a72541 [id=df94703d2472430387b81c028130d628]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.258952 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452f77091adc4cf09de2231f2f6d845b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.259239 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6661b9e9a174849b6093d09cd8c1251. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.262514 1630 tablet_service.cc:1505] Processing CreateTablet for tablet fca73ab61f0c47df9186459caf2b9a32 (DEFAULT_TABLE table=default.Table_dd9b9ef96622439ea016c3813fcbb81a [id=be1cc9d047554ea6a9afa531fb08a81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.263007 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca73ab61f0c47df9186459caf2b9a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.266904 1628 tablet_service.cc:1505] Processing CreateTablet for tablet d76410afd7ba4c73836f4791ef739056 (DEFAULT_TABLE table=default.Table_34bbddd692a44f46a4925c957f4e8848 [id=6f9b0ed7552c4556afd0f59316831a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.267342 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d76410afd7ba4c73836f4791ef739056. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.267654 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 3754540ae21e4c65b5af3d0aa3f9189f (DEFAULT_TABLE table=default.Table_eb1e63c7ef2244259c8a64fd4fe3859b [id=fec3ddedec0b44db80b30c899043bc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.268079 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3754540ae21e4c65b5af3d0aa3f9189f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.268574 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 0a12e415590a4c71aab4ad7d30412c83 (DEFAULT_TABLE table=default.Table_5a1d470fc7bc493393e47da7db62dfca [id=18ec057e920e4c21ae8cbe298058b6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.269132 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a12e415590a4c71aab4ad7d30412c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.274040 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 5b96fc2309274048bed7a35e4bf4bae2 (DEFAULT_TABLE table=default.Table_dc366a1f70d84d4f8276c1872b9fa0f7 [id=0f190205e9b347f6906d6f86b0624681]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.274833 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b96fc2309274048bed7a35e4bf4bae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.279111 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 0a12e415590a4c71aab4ad7d30412c83 (DEFAULT_TABLE table=default.Table_5a1d470fc7bc493393e47da7db62dfca [id=18ec057e920e4c21ae8cbe298058b6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.279584 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a12e415590a4c71aab4ad7d30412c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.286450 1339 tablet_service.cc:1505] Processing CreateTablet for tablet aebf76d4a66947ed9756643740418798 (DEFAULT_TABLE table=default.Table_48b3dfe737b54b9a8618c0a658e87e16 [id=665389c7bde54b7786edabcb91a9b33b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.286965 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aebf76d4a66947ed9756643740418798. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.295823 1617 tablet_service.cc:1505] Processing CreateTablet for tablet ad81d80697984ff1834a7d795e0f4b9b (DEFAULT_TABLE table=default.Table_61b7f6c428aa4bd9a65c1d66171c59c9 [id=6c604c6f86f944a285866376d9a5ce65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.296283 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad81d80697984ff1834a7d795e0f4b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.299575 1825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 19:47:58.300640 1825 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 19:47:58.300982 1825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 19:47:58.303488 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 037c8cc1e53447109fafc047c02f196f (DEFAULT_TABLE table=default.Table_4c8922124e774c8aa7e7a2db295099c3 [id=5bc052b3e0f04699b4b391bcec33617d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.304237 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 037c8cc1e53447109fafc047c02f196f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.310619 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 7503b22d9e12481b9af96385b9d76e50 (DEFAULT_TABLE table=default.Table_18392df3579642bf80ac127778fd508f [id=9065c680f4da42b5a6e5ca335c8e2f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.313412 1626 tablet_service.cc:1505] Processing CreateTablet for tablet a3259823b622461fa823841925b3d203 (DEFAULT_TABLE table=default.Table_d5ca9b9774254589b656ade4f08dc4c1 [id=350b37bf42194fcfbb660bffe83e47c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.314649 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet 90405a0c13bd4c229b011c90232f751d from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:58.315575 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet b9b1e0338fd14634a1e921735ae61289 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:58.316532 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 3754540ae21e4c65b5af3d0aa3f9189f (DEFAULT_TABLE table=default.Table_eb1e63c7ef2244259c8a64fd4fe3859b [id=fec3ddedec0b44db80b30c899043bc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.316917 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3754540ae21e4c65b5af3d0aa3f9189f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.317453 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e9bc5ade2870426abe90a5f3b20ebb5e" } new_table_name: "default.Table_ef3fdce598e74165bddac418dc92c20e" modify_external_catalogs: true
I20251024 19:47:58.317521 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 2037fcca87b94b5fb31639b4d07307b0 (DEFAULT_TABLE table=default.Table_9edfe580f57e43919a206badbc0bbfd6 [id=d28fd80220c64f8496a09f5d4e712874]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.317749 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_48b3dfe737b54b9a8618c0a658e87e16" } new_table_name: "default.Table_d09f149249c44c50adcb71321c133f2d" modify_external_catalogs: true
I20251024 19:47:58.319005 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet 897db285f5c84930a5b438be1cb4eb96 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:58.319643 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3259823b622461fa823841925b3d203. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.320533 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet 8721ee81e4e24153a0aad8ce99f872e1 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:58.321890 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 452f77091adc4cf09de2231f2f6d845b (DEFAULT_TABLE table=default.Table_573ec53c3d2c4de8864f399b291d8aa9 [id=9a49120c5b654676a63cd3966eaf31a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.322271 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452f77091adc4cf09de2231f2f6d845b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.323354 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7503b22d9e12481b9af96385b9d76e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.326859 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 5368789df6024d7daa9ccc2257137d58 (DEFAULT_TABLE table=default.Table_a81e7aae0a7a4f3db5c84efc4b16f873 [id=16bcec63f27d4dcd826ef11d45d58400]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.327234 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5368789df6024d7daa9ccc2257137d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.331915 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 73ffe342f541447d8519ee24bc98a0f3 (DEFAULT_TABLE table=default.Table_873d6a3a39344554b0ca73c575129c36 [id=508d6cdc5d4a494bad31eaf36383e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.332310 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 897db285f5c84930a5b438be1cb4eb96
I20251024 19:47:58.332839 1188 catalog_manager.cc:6719] ReplaceTablet: tablet 897db285f5c84930a5b438be1cb4eb96 deleted and replaced by tablet 758df87acbdb4adaa83e1ecbc44e7cf1
I20251024 19:47:58.333714 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 90405a0c13bd4c229b011c90232f751d
I20251024 19:47:58.334180 1185 catalog_manager.cc:6719] ReplaceTablet: tablet 90405a0c13bd4c229b011c90232f751d deleted and replaced by tablet 59c7dbe9ff2c4869994c4f43d5676c30
W20251024 19:47:58.337190 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4db17d9aff6542b6b94c09f872a4cceb consensus state has no leader
I20251024 19:47:58.339313 1491 tablet_service.cc:1505] Processing CreateTablet for tablet aebf76d4a66947ed9756643740418798 (DEFAULT_TABLE table=default.Table_48b3dfe737b54b9a8618c0a658e87e16 [id=665389c7bde54b7786edabcb91a9b33b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.339874 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aebf76d4a66947ed9756643740418798. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.340330 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet 897db285f5c84930a5b438be1cb4eb96 with delete_type TABLET_DATA_DELETED (replaced by tablet 758df87acbdb4adaa83e1ecbc44e7cf1) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.340677 1627 tablet_service.cc:1552] Processing DeleteTablet for tablet 90405a0c13bd4c229b011c90232f751d with delete_type TABLET_DATA_DELETED (replaced by tablet 59c7dbe9ff2c4869994c4f43d5676c30) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.342967 1768 tablet_bootstrap.cc:492] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:58.343786 1768 ts_tablet_manager.cc:1403] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.393s user 0.013s sys 0.003s
I20251024 19:47:58.344358 1767 tablet_bootstrap.cc:492] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:58.344601 1767 ts_tablet_manager.cc:1403] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.233s user 0.004s sys 0.014s
I20251024 19:47:58.345803 1768 raft_consensus.cc:359] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.349182 1768 raft_consensus.cc:385] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.349534 1775 ts_tablet_manager.cc:1916] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:58.349979 1768 raft_consensus.cc:740] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.352671 1767 raft_consensus.cc:359] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.352945 1767 raft_consensus.cc:385] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.353065 1767 raft_consensus.cc:740] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.353123 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cdf2e8872b724c7aa86d5091eb52fb1c" } modify_external_catalogs: true
I20251024 19:47:58.353305 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cdf2e8872b724c7aa86d5091eb52fb1c" } modify_external_catalogs: true
I20251024 19:47:58.353374 1767 consensus_queue.cc:260] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.355113 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a81e7aae0a7a4f3db5c84efc4b16f873" } modify_external_catalogs: true
I20251024 19:47:58.355293 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a81e7aae0a7a4f3db5c84efc4b16f873" } modify_external_catalogs: true
I20251024 19:47:58.356627 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d8ae96b95894421588b9b807f296a627"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.357205 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8ae96b95894421588b9b807f296a627 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.354082 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ffe342f541447d8519ee24bc98a0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.358968 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dbcb50e578964f0f9c6d722e4ff4e6a4" } new_table_name: "default.Table_9764ce0360fd4fcfb03ab7df33ed112a" modify_external_catalogs: true
I20251024 19:47:58.359273 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2037fcca87b94b5fb31639b4d07307b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.360386 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_eb1e63c7ef2244259c8a64fd4fe3859b" } modify_external_catalogs: true
I20251024 19:47:58.360561 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_eb1e63c7ef2244259c8a64fd4fe3859b" } modify_external_catalogs: true
I20251024 19:47:58.362433 1767 ts_tablet_manager.cc:1434] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.018s user 0.002s sys 0.001s
I20251024 19:47:58.366458 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_450ee0c26fcb4ab7961f7cbde38becb1" } new_table_name: "default.Table_bc5c06e7b159482c89d22461fe071b3f" modify_external_catalogs: true
I20251024 19:47:58.368649 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b9caeb0e904f44dc8f879c8988a34678" } new_table_name: "default.Table_d0ae44b1936f452d9c75bd4b888bfb30" modify_external_catalogs: true
I20251024 19:47:58.369998 1882 raft_consensus.cc:493] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:58.370190 1882 raft_consensus.cc:515] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.370285 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_52c56313187f4435b2136108330e12b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.370900 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52c56313187f4435b2136108330e12b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.370996 1882 leader_election.cc:290] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:58.379062 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 8721ee81e4e24153a0aad8ce99f872e1
I20251024 19:47:58.379518 1187 catalog_manager.cc:6719] ReplaceTablet: tablet 8721ee81e4e24153a0aad8ce99f872e1 deleted and replaced by tablet b79801594e95434cbb8fe7b75b14602e
I20251024 19:47:58.383863 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_be61dd09c4c94f568d723c4bace3a038"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.384776 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be61dd09c4c94f568d723c4bace3a038 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.385807 1774 ts_tablet_manager.cc:1929] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:58.385957 1774 log.cc:1199] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/f396c2982f8d487cb44c9422b09da14a
I20251024 19:47:58.386323 1774 ts_tablet_manager.cc:1950] T f396c2982f8d487cb44c9422b09da14a P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:58.387115 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 86) took 1437 ms. Trace:
I20251024 19:47:58.387228 1774 rpcz_store.cc:276] 1024 19:47:56.949561 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:56.949730 (+ 169us) service_pool.cc:225] Handling call
1024 19:47:58.194484 (+1244754us) ts_tablet_manager.cc:1151] Deleting tablet f396c2982f8d487cb44c9422b09da14a
1024 19:47:58.194494 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:58.313450 (+118956us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:58.385657 (+ 72207us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:58.387078 (+ 1421us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1244516}
I20251024 19:47:58.387585 1774 tablet_replica.cc:333] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:58.387799 1774 raft_consensus.cc:2243] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:58.387970 1774 raft_consensus.cc:2272] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:58.393020 1775 ts_tablet_manager.cc:1929] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251024 19:47:58.393712 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aebf76d4a66947ed9756643740418798 consensus state has no leader
I20251024 19:47:58.398779 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4c3de3c56f624eee8d86a60154ec6f07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.399418 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c3de3c56f624eee8d86a60154ec6f07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.399828 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 8721ee81e4e24153a0aad8ce99f872e1 with delete_type TABLET_DATA_DELETED (replaced by tablet b79801594e95434cbb8fe7b75b14602e) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.400218 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet e2d97caee586418a8981108465f6b704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.400602 1767 tablet_bootstrap.cc:492] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:58.402978 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ba85164015448d4a688a473cce90c5c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:58.402514 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ba85164015448d4a688a473cce90c5c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:47:58.403966 1308 leader_election.cc:343] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:58.404091 1775 log.cc:1199] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/fc8c01a4c07a42a0bc48a58313090e68
I20251024 19:47:58.404588 1775 ts_tablet_manager.cc:1950] T fc8c01a4c07a42a0bc48a58313090e68 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:58.406808 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f91316c030dc4f7da27977a9dde01f4b
W20251024 19:47:58.406999 1305 leader_election.cc:343] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:58.407168 1305 leader_election.cc:304] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:58.408071 1882 raft_consensus.cc:2749] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:58.410466 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4db7a2da5af4481db5b546d201dfd278" } modify_external_catalogs: true
I20251024 19:47:58.410739 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4db7a2da5af4481db5b546d201dfd278" } modify_external_catalogs: true
I20251024 19:47:58.412559 1768 consensus_queue.cc:260] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.416760 1766 tablet_bootstrap.cc:492] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:58.416909 1766 ts_tablet_manager.cc:1403] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.373s user 0.008s sys 0.004s
I20251024 19:47:58.417680 1766 raft_consensus.cc:359] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.417876 1766 raft_consensus.cc:385] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.417979 1766 raft_consensus.cc:740] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.421473 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet f91316c030dc4f7da27977a9dde01f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.424669 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 5368789df6024d7daa9ccc2257137d58
I20251024 19:47:58.427904 1776 ts_tablet_manager.cc:1929] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:58.429366 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_23a3d28e6ce941008c62ffbf100cc922"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.430243 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23a3d28e6ce941008c62ffbf100cc922 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.431087 1767 tablet_bootstrap.cc:654] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.431497 1768 ts_tablet_manager.cc:1434] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.087s user 0.002s sys 0.000s
I20251024 19:47:58.432190 1776 log.cc:1199] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/608696ec90ed45e1ad939ed2bfc88d34
I20251024 19:47:58.435868 1768 tablet_bootstrap.cc:492] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:58.436980 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet f396c2982f8d487cb44c9422b09da14a (table default.Table_69e288a9d6d24a5f841f62cd9a208f0f [id=969d428d45db42d18a5a9b82ec945bfb]) successfully deleted
I20251024 19:47:58.438129 1491 tablet_service.cc:1505] Processing CreateTablet for tablet ad055dc3387b414bb55d09b9683ba17e (DEFAULT_TABLE table=default.Table_97c6fc690615425aa5af0d8c0dd249af [id=c3de6937e93a48dcb88cabee59a09d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.438555 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad055dc3387b414bb55d09b9683ba17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.441097 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet 5368789df6024d7daa9ccc2257137d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.441799 1776 ts_tablet_manager.cc:1950] T 608696ec90ed45e1ad939ed2bfc88d34 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:58.441989 1511 raft_consensus.cc:1275] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Refusing update from remote peer af3952de050140d383f6b0de7666fa97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 19:47:58.444187 1887 consensus_queue.cc:1048] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251024 19:47:58.444464 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet c292ce055def4964ab143252c6b2b87e from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:58.450647 1776 tablet_replica.cc:333] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:58.451565 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 608696ec90ed45e1ad939ed2bfc88d34 (table default.Table_6e1fbc007b464e8c93803ba262a19fea [id=2c92cb09b287410c90b811e4b88c2260]) successfully deleted
W20251024 19:47:58.452276 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ba85164015448d4a688a473cce90c5c consensus state has no leader
I20251024 19:47:58.453735 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 5fd7726c532145c6b44ea9a37544510d (DEFAULT_TABLE table=default.Table_999546c6f492460bb6dc96ab55b7d3ad [id=7dbbe53e70a547888a510250e0583e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.454185 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fd7726c532145c6b44ea9a37544510d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.455039 1776 raft_consensus.cc:2243] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:58.455361 1776 raft_consensus.cc:2272] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:58.457316 1341 tablet_service.cc:1505] Processing CreateTablet for tablet ad055dc3387b414bb55d09b9683ba17e (DEFAULT_TABLE table=default.Table_97c6fc690615425aa5af0d8c0dd249af [id=c3de6937e93a48dcb88cabee59a09d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.458071 1371 raft_consensus.cc:1275] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Refusing update from remote peer af3952de050140d383f6b0de7666fa97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 19:47:58.458611 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad055dc3387b414bb55d09b9683ba17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.459512 1887 consensus_queue.cc:1048] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:47:58.459625 1767 tablet_bootstrap.cc:492] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:58.460728 1776 ts_tablet_manager.cc:1916] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:58.461199 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3754540ae21e4c65b5af3d0aa3f9189f
I20251024 19:47:58.461541 1767 ts_tablet_manager.cc:1403] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.003s
I20251024 19:47:58.464661 1767 raft_consensus.cc:359] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:58.466670 1767 raft_consensus.cc:385] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.466841 1767 raft_consensus.cc:740] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.467206 1767 consensus_queue.cc:260] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:47:58.473214 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe97e31310ae4733ba93de1b62a37842 consensus state has no leader
I20251024 19:47:58.463665 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 3754540ae21e4c65b5af3d0aa3f9189f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.474359 1767 ts_tablet_manager.cc:1434] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251024 19:47:58.476204 1766 consensus_queue.cc:260] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.477411 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 6ec5c7bf36874423b4790c6169da8bd0 (DEFAULT_TABLE table=default.Table_52ecaf891a2449d78bb8a354e7fb904b [id=e0a69533d0fa41a4ad58f08e510fa39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.477962 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec5c7bf36874423b4790c6169da8bd0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:58.479221 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 95d7fbfa8ae54da49dd5603294faa7d3 (table default.Table_52ec55c23c394294a68850ab6b33aabe [id=fb0e7cab20f44aaebdb09862bd193f41], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 95d7fbfa8ae54da49dd5603294faa7d3 consensus state has no leader
I20251024 19:47:58.479586 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet ae86e74f810c4b989ac633f59e6ac445 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:47:58.484491 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50d919a0b3d74e9a97d136a84d50e24b consensus state has no leader
I20251024 19:47:58.487339 1482 tablet_service.cc:1505] Processing CreateTablet for tablet ce0ef4443e574846a9f48fef3b893352 (DEFAULT_TABLE table=default.Table_ce25ea67811e4818be50de93930ab68e [id=ec57efb4d37e44f1b68045f58bc11c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.487787 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce0ef4443e574846a9f48fef3b893352. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.488447 1767 tablet_bootstrap.cc:492] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:58.490777 1486 tablet_service.cc:1505] Processing CreateTablet for tablet ca0527af718f494a8bbcc66e7b325bed (DEFAULT_TABLE table=default.Table_51982d999b0847ec87fc62a473782a5c [id=9a05bcf7ad5d430c9ef6d6e18a27808e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.491146 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca0527af718f494a8bbcc66e7b325bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.491159 1767 tablet_bootstrap.cc:654] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.496577 1776 ts_tablet_manager.cc:1929] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:58.496891 1776 log.cc:1199] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/fc8c01a4c07a42a0bc48a58313090e68
I20251024 19:47:58.497617 1776 ts_tablet_manager.cc:1950] T fc8c01a4c07a42a0bc48a58313090e68 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:58.499593 1486 tablet_service.cc:1505] Processing CreateTablet for tablet d692e1988d13460fb1d54e930e17fc9d (DEFAULT_TABLE table=default.Table_f46ed7e555b3432b95a3d9b829968b8e [id=67a0be12c85648bdb4515a096db21d3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.500046 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d692e1988d13460fb1d54e930e17fc9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.501380 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet fc8c01a4c07a42a0bc48a58313090e68 (table default.Table_401da5a2f0dc42bebcb8f9076d40a8b8 [id=d955ab5050fa4039b48b0c9235b687cd]) successfully deleted
I20251024 19:47:58.499987 1767 tablet_bootstrap.cc:492] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:58.502039 1767 ts_tablet_manager.cc:1403] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20251024 19:47:58.503834 1767 raft_consensus.cc:359] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:58.504257 1767 raft_consensus.cc:385] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.504626 1767 raft_consensus.cc:740] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.505154 1490 tablet_service.cc:1505] Processing CreateTablet for tablet b20a6845ae1645c0b51f096d471d4ccc (DEFAULT_TABLE table=default.Table_59612aa90b7540f6bf9cfc72e1f68fc1 [id=201e5232763a4940a1288a56a695f71c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.505703 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b20a6845ae1645c0b51f096d471d4ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.505075 1767 consensus_queue.cc:260] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:58.510941 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 1745bbc68d6846fab1a6197f898754b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 31647a48fb834c80b6920c756a94250d) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.512636 1767 ts_tablet_manager.cc:1434] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.010s user 0.002s sys 0.002s
I20251024 19:47:58.513530 1766 ts_tablet_manager.cc:1434] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.097s user 0.003s sys 0.000s
W20251024 19:47:58.513499 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b5473699c1b647f8b86ee43fdb56bdfd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5473699c1b647f8b86ee43fdb56bdfd already in progress: creating tablet
W20251024 19:47:58.513844 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 897db285f5c84930a5b438be1cb4eb96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 897db285f5c84930a5b438be1cb4eb96 already in progress: creating tablet
I20251024 19:47:58.514496 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 9fffc42ddda14bd28800917ff2f0ced5 (DEFAULT_TABLE table=default.Table_64aaab6eda0a443fbb49223835af43e3 [id=52efb5f413d14db28314ecbb058a351f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.514998 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fffc42ddda14bd28800917ff2f0ced5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.516611 1767 tablet_bootstrap.cc:492] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:58.522325 1489 tablet_service.cc:1505] Processing CreateTablet for tablet b6661b9e9a174849b6093d09cd8c1251 (DEFAULT_TABLE table=default.Table_7dbe5740202642f1be406c4897a72541 [id=df94703d2472430387b81c028130d628]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.522794 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6661b9e9a174849b6093d09cd8c1251. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.531440 1766 tablet_bootstrap.cc:492] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:47:58.532706 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 90405a0c13bd4c229b011c90232f751d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90405a0c13bd4c229b011c90232f751d already in progress: creating tablet
I20251024 19:47:58.533809 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet c292ce055def4964ab143252c6b2b87e
I20251024 19:47:58.534446 1184 catalog_manager.cc:6719] ReplaceTablet: tablet c292ce055def4964ab143252c6b2b87e deleted and replaced by tablet d9bf08ccf7054fedb9efcbb00354c71a
I20251024 19:47:58.534479 1776 tablet_replica.cc:333] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:58.534723 1776 raft_consensus.cc:2243] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:58.534948 1776 raft_consensus.cc:2272] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:58.537195 1767 tablet_bootstrap.cc:654] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:47:58.538504 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 8721ee81e4e24153a0aad8ce99f872e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8721ee81e4e24153a0aad8ce99f872e1 already in progress: creating tablet
I20251024 19:47:58.539201 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet c292ce055def4964ab143252c6b2b87e with delete_type TABLET_DATA_DELETED (replaced by tablet d9bf08ccf7054fedb9efcbb00354c71a) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.539624 1766 tablet_bootstrap.cc:654] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.532624 1489 tablet_service.cc:1505] Processing CreateTablet for tablet fca73ab61f0c47df9186459caf2b9a32 (DEFAULT_TABLE table=default.Table_dd9b9ef96622439ea016c3813fcbb81a [id=be1cc9d047554ea6a9afa531fb08a81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.540987 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca73ab61f0c47df9186459caf2b9a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.542870 1776 ts_tablet_manager.cc:1916] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:58.546492 1484 tablet_service.cc:1505] Processing CreateTablet for tablet d76410afd7ba4c73836f4791ef739056 (DEFAULT_TABLE table=default.Table_34bbddd692a44f46a4925c957f4e8848 [id=6f9b0ed7552c4556afd0f59316831a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.546944 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d76410afd7ba4c73836f4791ef739056. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.547957 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 5b96fc2309274048bed7a35e4bf4bae2 (DEFAULT_TABLE table=default.Table_dc366a1f70d84d4f8276c1872b9fa0f7 [id=0f190205e9b347f6906d6f86b0624681]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.548444 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b96fc2309274048bed7a35e4bf4bae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.549782 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 (DEFAULT_TABLE table=default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.550159 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758df87acbdb4adaa83e1ecbc44e7cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.550477 1882 raft_consensus.cc:493] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:58.550740 1882 raft_consensus.cc:515] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.551560 1882 leader_election.cc:290] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:58.552421 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "047436af5f6d4f688bd83e33089d3ee4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:58.552762 1510 raft_consensus.cc:2468] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:47:58.553532 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 59c7dbe9ff2c4869994c4f43d5676c30 (DEFAULT_TABLE table=default.Table_eb9c8019451d46fba31a87a0b6ac6ad8 [id=e5b1dd31c2114e8a95ba33981029255c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.554106 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c7dbe9ff2c4869994c4f43d5676c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.555387 1308 leader_election.cc:304] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:47:58.555739 1882 raft_consensus.cc:2804] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:47:58.555858 1882 raft_consensus.cc:493] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:47:58.555938 1882 raft_consensus.cc:3060] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:58.560114 1342 tablet_service.cc:1505] Processing CreateTablet for tablet ce0ef4443e574846a9f48fef3b893352 (DEFAULT_TABLE table=default.Table_ce25ea67811e4818be50de93930ab68e [id=ec57efb4d37e44f1b68045f58bc11c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.560515 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 6ec5c7bf36874423b4790c6169da8bd0 (DEFAULT_TABLE table=default.Table_52ecaf891a2449d78bb8a354e7fb904b [id=e0a69533d0fa41a4ad58f08e510fa39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.560820 1348 tablet_service.cc:1505] Processing CreateTablet for tablet ca0527af718f494a8bbcc66e7b325bed (DEFAULT_TABLE table=default.Table_51982d999b0847ec87fc62a473782a5c [id=9a05bcf7ad5d430c9ef6d6e18a27808e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.561154 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 5fd7726c532145c6b44ea9a37544510d (DEFAULT_TABLE table=default.Table_999546c6f492460bb6dc96ab55b7d3ad [id=7dbbe53e70a547888a510250e0583e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.562388 1351 tablet_service.cc:1505] Processing CreateTablet for tablet d76410afd7ba4c73836f4791ef739056 (DEFAULT_TABLE table=default.Table_34bbddd692a44f46a4925c957f4e8848 [id=6f9b0ed7552c4556afd0f59316831a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.562525 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 9fffc42ddda14bd28800917ff2f0ced5 (DEFAULT_TABLE table=default.Table_64aaab6eda0a443fbb49223835af43e3 [id=52efb5f413d14db28314ecbb058a351f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.562634 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "047436af5f6d4f688bd83e33089d3ee4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:58.562922 1352 tablet_service.cc:1505] Processing CreateTablet for tablet b6661b9e9a174849b6093d09cd8c1251 (DEFAULT_TABLE table=default.Table_7dbe5740202642f1be406c4897a72541 [id=df94703d2472430387b81c028130d628]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:47:58.563262 1305 leader_election.cc:343] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:58.563458 1340 tablet_service.cc:1505] Processing CreateTablet for tablet fca73ab61f0c47df9186459caf2b9a32 (DEFAULT_TABLE table=default.Table_dd9b9ef96622439ea016c3813fcbb81a [id=be1cc9d047554ea6a9afa531fb08a81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.564088 1346 tablet_service.cc:1505] Processing CreateTablet for tablet d692e1988d13460fb1d54e930e17fc9d (DEFAULT_TABLE table=default.Table_f46ed7e555b3432b95a3d9b829968b8e [id=67a0be12c85648bdb4515a096db21d3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.564404 1339 tablet_service.cc:1505] Processing CreateTablet for tablet b20a6845ae1645c0b51f096d471d4ccc (DEFAULT_TABLE table=default.Table_59612aa90b7540f6bf9cfc72e1f68fc1 [id=201e5232763a4940a1288a56a695f71c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.564805 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce0ef4443e574846a9f48fef3b893352. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.565244 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d76410afd7ba4c73836f4791ef739056. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.565526 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b20a6845ae1645c0b51f096d471d4ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.565963 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fffc42ddda14bd28800917ff2f0ced5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.566097 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 030e690ce4df4818a6450ae293b0ffbd
I20251024 19:47:58.567137 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 030e690ce4df4818a6450ae293b0ffbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.569779 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec5c7bf36874423b4790c6169da8bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.573720 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca73ab61f0c47df9186459caf2b9a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.577261 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d692e1988d13460fb1d54e930e17fc9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.581410 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6661b9e9a174849b6093d09cd8c1251. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.585644 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca0527af718f494a8bbcc66e7b325bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.592839 1767 tablet_bootstrap.cc:492] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:58.593287 1767 ts_tablet_manager.cc:1403] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.077s user 0.000s sys 0.006s
I20251024 19:47:58.595024 1767 raft_consensus.cc:359] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.595556 1767 raft_consensus.cc:385] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.596295 1767 raft_consensus.cc:740] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.599241 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fd7726c532145c6b44ea9a37544510d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.598834 1767 consensus_queue.cc:260] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.610459 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 047436af5f6d4f688bd83e33089d3ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.609695 1767 ts_tablet_manager.cc:1434] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20251024 19:47:58.616475 1767 tablet_bootstrap.cc:492] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:58.618122 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 187925082cdd4923a6e9148bf9f270fd (DEFAULT_TABLE table=default.Table_23a3d28e6ce941008c62ffbf100cc922 [id=33f5ac73298543d0b5af77a27e897b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.618948 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 187925082cdd4923a6e9148bf9f270fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.565986 1882 raft_consensus.cc:515] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.625592 1882 leader_election.cc:290] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:58.626072 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "047436af5f6d4f688bd83e33089d3ee4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:47:58.626206 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "047436af5f6d4f688bd83e33089d3ee4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:47:58.626402 1510 raft_consensus.cc:3060] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:58.626448 1648 raft_consensus.cc:3060] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:58.630200 1510 raft_consensus.cc:2468] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:47:58.630470 1648 raft_consensus.cc:2468] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:47:58.630795 1308 leader_election.cc:304] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:47:58.631322 1882 raft_consensus.cc:2804] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:47:58.633301 1623 tablet_service.cc:1505] Processing CreateTablet for tablet dc14b0cdb7f342ac956a7857ab3f35d7 (DEFAULT_TABLE table=default.Table_be61dd09c4c94f568d723c4bace3a038 [id=66be3fc672ea4a9e9fb5dde3ae5e5b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.633591 1882 raft_consensus.cc:697] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:47:58.633869 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc14b0cdb7f342ac956a7857ab3f35d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.634089 1882 consensus_queue.cc:237] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.634495 1768 tablet_bootstrap.cc:654] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.642248 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 7352d84226db467dad31ae5941842b80 (DEFAULT_TABLE table=default.Table_52c56313187f4435b2136108330e12b4 [id=d115a3523a5746918a343221a41c9589]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.642808 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7352d84226db467dad31ae5941842b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.646322 1625 tablet_service.cc:1505] Processing CreateTablet for tablet b79801594e95434cbb8fe7b75b14602e (DEFAULT_TABLE table=default.Table_9291fbfe8e484be094f28ad94814b13f [id=7ec9391c52b1424eb52850f2c6ba4779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.646601 1626 tablet_service.cc:1505] Processing CreateTablet for tablet fdfa5eea3b674a3389a156d1c72204df (DEFAULT_TABLE table=default.Table_d8ae96b95894421588b9b807f296a627 [id=96ea5a86f92b4d18b0d74f1092bd17b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.647476 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 635a132735704cfda69bf7c8deaea81d (DEFAULT_TABLE table=default.Table_4c3de3c56f624eee8d86a60154ec6f07 [id=93a61862707b4a7ba8a0ff349704dc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.646723 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b79801594e95434cbb8fe7b75b14602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.648077 1768 tablet_bootstrap.cc:492] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:58.648377 1768 ts_tablet_manager.cc:1403] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.212s user 0.004s sys 0.000s
I20251024 19:47:58.649963 1768 raft_consensus.cc:359] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.650233 1768 raft_consensus.cc:385] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.650328 1768 raft_consensus.cc:740] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.650696 1768 consensus_queue.cc:260] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.653194 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 635a132735704cfda69bf7c8deaea81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.653719 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdfa5eea3b674a3389a156d1c72204df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.653825 1768 ts_tablet_manager.cc:1434] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251024 19:47:58.654953 1768 tablet_bootstrap.cc:492] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:58.657231 1768 tablet_bootstrap.cc:654] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.662158 1624 tablet_service.cc:1505] Processing CreateTablet for tablet d9bf08ccf7054fedb9efcbb00354c71a (DEFAULT_TABLE table=default.Table_5e63f20b2f66459795a5289f47ffaf0e [id=10833938cf88466b834698420d7c01e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.662642 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9bf08ccf7054fedb9efcbb00354c71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.685146 1768 tablet_bootstrap.cc:492] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:58.685334 1768 ts_tablet_manager.cc:1403] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20251024 19:47:58.686414 1768 raft_consensus.cc:359] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.686664 1768 raft_consensus.cc:385] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.686762 1768 raft_consensus.cc:740] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.687112 1768 consensus_queue.cc:260] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.688843 1768 ts_tablet_manager.cc:1434] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251024 19:47:58.689991 1768 tablet_bootstrap.cc:492] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:58.692058 1768 tablet_bootstrap.cc:654] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.702708 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 5b96fc2309274048bed7a35e4bf4bae2 (DEFAULT_TABLE table=default.Table_dc366a1f70d84d4f8276c1872b9fa0f7 [id=0f190205e9b347f6906d6f86b0624681]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.703106 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b96fc2309274048bed7a35e4bf4bae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.703776 1483 tablet_service.cc:1505] Processing CreateTablet for tablet ad81d80697984ff1834a7d795e0f4b9b (DEFAULT_TABLE table=default.Table_61b7f6c428aa4bd9a65c1d66171c59c9 [id=6c604c6f86f944a285866376d9a5ce65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.704385 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad81d80697984ff1834a7d795e0f4b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.714821 1340 tablet_service.cc:1505] Processing CreateTablet for tablet ad81d80697984ff1834a7d795e0f4b9b (DEFAULT_TABLE table=default.Table_61b7f6c428aa4bd9a65c1d66171c59c9 [id=6c604c6f86f944a285866376d9a5ce65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.715287 1768 tablet_bootstrap.cc:492] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:58.715394 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad81d80697984ff1834a7d795e0f4b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.715480 1768 ts_tablet_manager.cc:1403] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20251024 19:47:58.716354 1768 raft_consensus.cc:359] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.716600 1768 raft_consensus.cc:385] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.716707 1768 raft_consensus.cc:740] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.717046 1768 consensus_queue.cc:260] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.718602 1768 ts_tablet_manager.cc:1434] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251024 19:47:58.719161 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 7503b22d9e12481b9af96385b9d76e50 (DEFAULT_TABLE table=default.Table_18392df3579642bf80ac127778fd508f [id=9065c680f4da42b5a6e5ca335c8e2f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.719812 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7503b22d9e12481b9af96385b9d76e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.720023 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 7503b22d9e12481b9af96385b9d76e50 (DEFAULT_TABLE table=default.Table_18392df3579642bf80ac127778fd508f [id=9065c680f4da42b5a6e5ca335c8e2f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.720422 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7503b22d9e12481b9af96385b9d76e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.720408 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 2037fcca87b94b5fb31639b4d07307b0 (DEFAULT_TABLE table=default.Table_9edfe580f57e43919a206badbc0bbfd6 [id=d28fd80220c64f8496a09f5d4e712874]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.720901 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2037fcca87b94b5fb31639b4d07307b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.723326 1351 tablet_service.cc:1505] Processing CreateTablet for tablet a3259823b622461fa823841925b3d203 (DEFAULT_TABLE table=default.Table_d5ca9b9774254589b656ade4f08dc4c1 [id=350b37bf42194fcfbb660bffe83e47c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.723802 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3259823b622461fa823841925b3d203. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.723850 1488 tablet_service.cc:1505] Processing CreateTablet for tablet a3259823b622461fa823841925b3d203 (DEFAULT_TABLE table=default.Table_d5ca9b9774254589b656ade4f08dc4c1 [id=350b37bf42194fcfbb660bffe83e47c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.724372 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3259823b622461fa823841925b3d203. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.724651 1481 tablet_service.cc:1552] Processing DeleteTablet for tablet 897db285f5c84930a5b438be1cb4eb96 with delete_type TABLET_DATA_DELETED (replaced by tablet 758df87acbdb4adaa83e1ecbc44e7cf1) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:58.729059 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 2037fcca87b94b5fb31639b4d07307b0 (DEFAULT_TABLE table=default.Table_9edfe580f57e43919a206badbc0bbfd6 [id=d28fd80220c64f8496a09f5d4e712874]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.729686 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2037fcca87b94b5fb31639b4d07307b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.730408 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 897db285f5c84930a5b438be1cb4eb96 with delete_type TABLET_DATA_DELETED (replaced by tablet 758df87acbdb4adaa83e1ecbc44e7cf1) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:58.731011 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 90405a0c13bd4c229b011c90232f751d with delete_type TABLET_DATA_DELETED (replaced by tablet 59c7dbe9ff2c4869994c4f43d5676c30) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:58.731317 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 8721ee81e4e24153a0aad8ce99f872e1 with delete_type TABLET_DATA_DELETED (replaced by tablet b79801594e95434cbb8fe7b75b14602e) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:58.732060 1768 tablet_bootstrap.cc:492] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:58.733879 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 73ffe342f541447d8519ee24bc98a0f3 (DEFAULT_TABLE table=default.Table_873d6a3a39344554b0ca73c575129c36 [id=508d6cdc5d4a494bad31eaf36383e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.743155 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ffe342f541447d8519ee24bc98a0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.745946 1768 tablet_bootstrap.cc:654] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.746482 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 73ffe342f541447d8519ee24bc98a0f3 (DEFAULT_TABLE table=default.Table_873d6a3a39344554b0ca73c575129c36 [id=508d6cdc5d4a494bad31eaf36383e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.747018 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ffe342f541447d8519ee24bc98a0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.729549 1774 ts_tablet_manager.cc:1916] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:58.748349 1481 tablet_service.cc:1552] Processing DeleteTablet for tablet 90405a0c13bd4c229b011c90232f751d with delete_type TABLET_DATA_DELETED (replaced by tablet 59c7dbe9ff2c4869994c4f43d5676c30) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:58.750056 1480 tablet_service.cc:1552] Processing DeleteTablet for tablet 8721ee81e4e24153a0aad8ce99f872e1 with delete_type TABLET_DATA_DELETED (replaced by tablet b79801594e95434cbb8fe7b75b14602e) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:58.762658 1767 tablet_bootstrap.cc:654] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:47:58.769798 1305 consensus_peers.cc:597] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 922f975a672746b3b72acf4d3050f688. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:47:58.783154 1192 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet f1ad65fd080e452e8094975d420336c7 (table default.Table_a54ffafa18b848b79a290f58c9754bec [id=2fd5c541b06a46469e65ed0942d2c394]). Expected version 4 got 0
I20251024 19:47:58.773628 1776 ts_tablet_manager.cc:1929] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:58.783444 1776 log.cc:1199] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/e2d97caee586418a8981108465f6b704
I20251024 19:47:58.783983 1776 ts_tablet_manager.cc:1950] T e2d97caee586418a8981108465f6b704 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:58.785243 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet e2d97caee586418a8981108465f6b704 (table default.Table_95644b1e2fa0449d806efb9c458266b9 [id=168a2adf03184a04b7e8f5017919a08c]) successfully deleted
W20251024 19:47:58.785822 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f91316c030dc4f7da27977a9dde01f4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f91316c030dc4f7da27977a9dde01f4b already in progress: creating tablet
W20251024 19:47:58.786315 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 5368789df6024d7daa9ccc2257137d58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5368789df6024d7daa9ccc2257137d58 already in progress: creating tablet
W20251024 19:47:58.786968 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3754540ae21e4c65b5af3d0aa3f9189f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3754540ae21e4c65b5af3d0aa3f9189f already in progress: creating tablet
I20251024 19:47:58.787238 1776 tablet_replica.cc:333] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:58.787768 1776 raft_consensus.cc:2243] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:58.788172 1776 raft_consensus.cc:2272] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:58.789691 1776 ts_tablet_manager.cc:1916] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:58.794816 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet f91316c030dc4f7da27977a9dde01f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:58.799795 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 5368789df6024d7daa9ccc2257137d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:58.802575 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 047436af5f6d4f688bd83e33089d3ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:58.803961 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet fb17a35d8d8446cf9870d05e107ed237 (table default.Table_51982d999b0847ec87fc62a473782a5c [id=9a05bcf7ad5d430c9ef6d6e18a27808e]) successfully deleted
W20251024 19:47:58.804538 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3caecdff61ce48f7b7c3846bb380d21e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3caecdff61ce48f7b7c3846bb380d21e already in progress: creating tablet
I20251024 19:47:58.806423 1767 tablet_bootstrap.cc:492] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:58.806783 1767 ts_tablet_manager.cc:1403] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.192s user 0.005s sys 0.003s
I20251024 19:47:58.811182 1955 tablet.cc:1722] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251024 19:47:58.815131 1192 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet f1ad65fd080e452e8094975d420336c7 (table default.Table_a54ffafa18b848b79a290f58c9754bec [id=2fd5c541b06a46469e65ed0942d2c394]). Expected version 4 got 0
I20251024 19:47:58.808583 1767 raft_consensus.cc:359] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.820185 1767 raft_consensus.cc:385] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.822372 1767 raft_consensus.cc:740] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
W20251024 19:47:58.826054 1308 consensus_peers.cc:597] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 922f975a672746b3b72acf4d3050f688. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:47:58.830511 1767 consensus_queue.cc:260] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.837966 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet fc8c01a4c07a42a0bc48a58313090e68 (table default.Table_401da5a2f0dc42bebcb8f9076d40a8b8 [id=d955ab5050fa4039b48b0c9235b687cd]) successfully deleted
I20251024 19:47:58.842952 1192 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet f1ad65fd080e452e8094975d420336c7 (table default.Table_a54ffafa18b848b79a290f58c9754bec [id=2fd5c541b06a46469e65ed0942d2c394]). Expected version 4 got 0
I20251024 19:47:58.844538 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 5368789df6024d7daa9ccc2257137d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:58.846633 1766 tablet_bootstrap.cc:492] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:58.847961 1766 ts_tablet_manager.cc:1403] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.316s user 0.004s sys 0.004s
W20251024 19:47:58.842423 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2e956ea914c545d79be6d632bfc4ec57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e956ea914c545d79be6d632bfc4ec57 already in progress: creating tablet
W20251024 19:47:58.849529 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b2ea545b2cc14f589eda9419e55234bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2ea545b2cc14f589eda9419e55234bf already in progress: creating tablet
W20251024 19:47:58.849979 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet f346a90a9efe442f82750336e3893deb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f346a90a9efe442f82750336e3893deb already in progress: creating tablet
W20251024 19:47:58.850487 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b5473699c1b647f8b86ee43fdb56bdfd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5473699c1b647f8b86ee43fdb56bdfd already in progress: creating tablet
I20251024 19:47:58.851604 1766 raft_consensus.cc:359] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:58.851995 1766 raft_consensus.cc:385] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.852138 1766 raft_consensus.cc:740] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.852530 1766 consensus_queue.cc:260] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:58.853307 1766 ts_tablet_manager.cc:1434] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251024 19:47:58.854456 1766 tablet_bootstrap.cc:492] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:47:58.857358 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 897db285f5c84930a5b438be1cb4eb96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 897db285f5c84930a5b438be1cb4eb96 already in progress: creating tablet
I20251024 19:47:58.857949 1768 tablet_bootstrap.cc:492] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:58.858206 1768 ts_tablet_manager.cc:1403] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.139s user 0.006s sys 0.006s
I20251024 19:47:58.861035 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet f91316c030dc4f7da27977a9dde01f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:47:58.862747 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 90405a0c13bd4c229b011c90232f751d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90405a0c13bd4c229b011c90232f751d already in progress: creating tablet
W20251024 19:47:58.865543 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 8721ee81e4e24153a0aad8ce99f872e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8721ee81e4e24153a0aad8ce99f872e1 already in progress: creating tablet
I20251024 19:47:58.867017 1768 raft_consensus.cc:359] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.867363 1768 raft_consensus.cc:385] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.868216 1768 raft_consensus.cc:740] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.868698 1768 consensus_queue.cc:260] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.881662 1768 ts_tablet_manager.cc:1434] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
W20251024 19:47:58.869021 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 5368789df6024d7daa9ccc2257137d58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5368789df6024d7daa9ccc2257137d58 already in progress: creating tablet
I20251024 19:47:58.885847 1776 ts_tablet_manager.cc:1929] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251024 19:47:58.887248 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet f91316c030dc4f7da27977a9dde01f4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f91316c030dc4f7da27977a9dde01f4b already in progress: creating tablet
I20251024 19:47:58.887854 1882 raft_consensus.cc:493] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:58.888008 1767 ts_tablet_manager.cc:1434] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.079s user 0.002s sys 0.002s
I20251024 19:47:58.888425 1767 tablet_bootstrap.cc:492] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:58.891134 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_46251d3ce016440287b2a378eae2e9a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:58.891611 1767 tablet_bootstrap.cc:654] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:47:58.891777 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46251d3ce016440287b2a378eae2e9a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.892997 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f96cb6877d844a50aacb91514a539fee" } modify_external_catalogs: true
I20251024 19:47:58.893194 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f96cb6877d844a50aacb91514a539fee" } modify_external_catalogs: true
I20251024 19:47:58.894611 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4d53ff8bfef04c2f82bef077ab04d4a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.895108 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d53ff8bfef04c2f82bef077ab04d4a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.895174 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 3754540ae21e4c65b5af3d0aa3f9189f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:58.895258 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7dc53b918ca7443b807133df217c0f75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.895780 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dc53b918ca7443b807133df217c0f75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.895860 1882 raft_consensus.cc:515] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.896554 1882 leader_election.cc:290] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:58.898048 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 97c6dce647064344b0b53a7ffaeafb75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.898572 1768 tablet_bootstrap.cc:492] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:58.898554 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6eb10ae51f3a43c2ab15f362708fba94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:58.899008 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6eb10ae51f3a43c2ab15f362708fba94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.900373 1768 tablet_bootstrap.cc:654] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.901067 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet c292ce055def4964ab143252c6b2b87e with delete_type TABLET_DATA_DELETED (replaced by tablet d9bf08ccf7054fedb9efcbb00354c71a) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:58.901824 1956 tablet.cc:1722] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251024 19:47:58.903061 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97c6dce647064344b0b53a7ffaeafb75" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:58.903347 1648 raft_consensus.cc:2468] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:47:58.904311 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97c6dce647064344b0b53a7ffaeafb75" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:58.905400 1776 log.cc:1199] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/ae86e74f810c4b989ac633f59e6ac445
W20251024 19:47:58.905355 1308 leader_election.cc:343] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:58.905458 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 3754540ae21e4c65b5af3d0aa3f9189f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:58.905886 1776 ts_tablet_manager.cc:1950] T ae86e74f810c4b989ac633f59e6ac445 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:58.906338 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_31d09f80ce0a4953b4764b8425875f89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:58.906880 1774 ts_tablet_manager.cc:1929] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:58.906914 1776 tablet_replica.cc:333] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:47:58.906961 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31d09f80ce0a4953b4764b8425875f89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:58.907034 1774 log.cc:1199] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/0397637f0778442a871528d5dd173689
I20251024 19:47:58.907114 1776 raft_consensus.cc:2243] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:58.907313 1776 raft_consensus.cc:2272] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:58.907372 1305 leader_election.cc:304] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
I20251024 19:47:58.907449 1774 ts_tablet_manager.cc:1950] T 0397637f0778442a871528d5dd173689 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:58.908281 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet ae86e74f810c4b989ac633f59e6ac445 (table default.Table_ea60f0e4363c406d9ccff5f16b8ecce9 [id=071606963978424489f30719fa42baa3]) successfully deleted
I20251024 19:47:58.908387 1912 mvcc.cc:204] Tried to move back new op lower bound from 7214429300481179648 to 7214429297948782592. Current Snapshot: MvccSnapshot[applied={T|T < 7214429300481179648}]
I20251024 19:47:58.912442 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet a48249f1824a40b087b80a563816d485
I20251024 19:47:58.913677 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet a48249f1824a40b087b80a563816d485 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:58.914016 1955 tablet.cc:1717] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97: Skipping requested alter to schema version 4, tablet already version 4
I20251024 19:47:58.919755 1767 tablet_bootstrap.cc:492] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:58.919965 1767 ts_tablet_manager.cc:1403] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.002s
I20251024 19:47:58.920996 1882 raft_consensus.cc:2804] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:47:58.920995 1767 raft_consensus.cc:359] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.921139 1882 raft_consensus.cc:493] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:47:58.921221 1767 raft_consensus.cc:385] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.921303 1882 raft_consensus.cc:3060] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:58.921331 1767 raft_consensus.cc:740] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.921639 1767 consensus_queue.cc:260] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.921738 1766 tablet_bootstrap.cc:654] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.923027 1767 ts_tablet_manager.cc:1434] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251024 19:47:58.923377 1767 tablet_bootstrap.cc:492] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:58.925307 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet c292ce055def4964ab143252c6b2b87e with delete_type TABLET_DATA_DELETED (replaced by tablet d9bf08ccf7054fedb9efcbb00354c71a) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:58.928463 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 59c7dbe9ff2c4869994c4f43d5676c30 (DEFAULT_TABLE table=default.Table_eb9c8019451d46fba31a87a0b6ac6ad8 [id=e5b1dd31c2114e8a95ba33981029255c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.929006 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c7dbe9ff2c4869994c4f43d5676c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.932106 1966 tablet.cc:1722] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251024 19:47:58.932550 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 030e690ce4df4818a6450ae293b0ffbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:58.933830 1767 tablet_bootstrap.cc:654] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.933909 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 (DEFAULT_TABLE table=default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.938172 1767 tablet_bootstrap.cc:492] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:58.938551 1767 ts_tablet_manager.cc:1403] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20251024 19:47:58.939926 1767 raft_consensus.cc:359] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.940220 1767 raft_consensus.cc:385] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.940423 1767 raft_consensus.cc:740] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.940750 1767 consensus_queue.cc:260] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.927407 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 030e690ce4df4818a6450ae293b0ffbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:58.941381 1767 ts_tablet_manager.cc:1434] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:47:58.935576 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 187925082cdd4923a6e9148bf9f270fd (DEFAULT_TABLE table=default.Table_23a3d28e6ce941008c62ffbf100cc922 [id=33f5ac73298543d0b5af77a27e897b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.936050 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 7352d84226db467dad31ae5941842b80 (DEFAULT_TABLE table=default.Table_52c56313187f4435b2136108330e12b4 [id=d115a3523a5746918a343221a41c9589]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.941715 1767 tablet_bootstrap.cc:492] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:58.936571 1348 tablet_service.cc:1505] Processing CreateTablet for tablet fdfa5eea3b674a3389a156d1c72204df (DEFAULT_TABLE table=default.Table_d8ae96b95894421588b9b807f296a627 [id=96ea5a86f92b4d18b0d74f1092bd17b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.936950 1343 tablet_service.cc:1505] Processing CreateTablet for tablet dc14b0cdb7f342ac956a7857ab3f35d7 (DEFAULT_TABLE table=default.Table_be61dd09c4c94f568d723c4bace3a038 [id=66be3fc672ea4a9e9fb5dde3ae5e5b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.925891 1882 raft_consensus.cc:515] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.943421 1767 tablet_bootstrap.cc:654] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.943701 1776 ts_tablet_manager.cc:1916] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:58.949765 1184 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet f1ad65fd080e452e8094975d420336c7 (table default.Table_a54ffafa18b848b79a290f58c9754bec [id=2fd5c541b06a46469e65ed0942d2c394]). Expected version 4 got 0
I20251024 19:47:58.957811 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4c8922124e774c8aa7e7a2db295099c3" } new_table_name: "default.Table_e87be853efba49aeb50e71270beac600" modify_external_catalogs: true
I20251024 19:47:58.961997 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2e71745e53a04c5b8701ff72aedb203c" } new_table_name: "default.Table_b15672d0c13d4c939f61e30a2feb96e6" modify_external_catalogs: true
W20251024 19:47:58.962023 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3754540ae21e4c65b5af3d0aa3f9189f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3754540ae21e4c65b5af3d0aa3f9189f already in progress: creating tablet
W20251024 19:47:58.962929 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet c292ce055def4964ab143252c6b2b87e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c292ce055def4964ab143252c6b2b87e already in progress: creating tablet
W20251024 19:47:58.963816 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 030e690ce4df4818a6450ae293b0ffbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 030e690ce4df4818a6450ae293b0ffbd already in progress: creating tablet
I20251024 19:47:58.967823 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 635a132735704cfda69bf7c8deaea81d (DEFAULT_TABLE table=default.Table_4c3de3c56f624eee8d86a60154ec6f07 [id=93a61862707b4a7ba8a0ff349704dc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.968226 1767 tablet_bootstrap.cc:492] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:58.968911 1767 ts_tablet_manager.cc:1403] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.015s
I20251024 19:47:58.971482 1767 raft_consensus.cc:359] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.972046 1767 raft_consensus.cc:385] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:58.973160 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758df87acbdb4adaa83e1ecbc44e7cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.972270 1767 raft_consensus.cc:740] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:58.975162 1767 consensus_queue.cc:260] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:58.980162 1767 ts_tablet_manager.cc:1434] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20251024 19:47:58.982383 1767 tablet_bootstrap.cc:492] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:58.985400 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 187925082cdd4923a6e9148bf9f270fd (DEFAULT_TABLE table=default.Table_23a3d28e6ce941008c62ffbf100cc922 [id=33f5ac73298543d0b5af77a27e897b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.985846 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 187925082cdd4923a6e9148bf9f270fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.986203 1767 tablet_bootstrap.cc:654] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:58.991303 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 7352d84226db467dad31ae5941842b80 (DEFAULT_TABLE table=default.Table_52c56313187f4435b2136108330e12b4 [id=d115a3523a5746918a343221a41c9589]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.991770 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7352d84226db467dad31ae5941842b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.993942 1488 tablet_service.cc:1505] Processing CreateTablet for tablet b79801594e95434cbb8fe7b75b14602e (DEFAULT_TABLE table=default.Table_9291fbfe8e484be094f28ad94814b13f [id=7ec9391c52b1424eb52850f2c6ba4779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.994379 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b79801594e95434cbb8fe7b75b14602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.995772 1338 tablet_service.cc:1505] Processing CreateTablet for tablet d9bf08ccf7054fedb9efcbb00354c71a (DEFAULT_TABLE table=default.Table_5e63f20b2f66459795a5289f47ffaf0e [id=10833938cf88466b834698420d7c01e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.998147 1491 tablet_service.cc:1505] Processing CreateTablet for tablet d9bf08ccf7054fedb9efcbb00354c71a (DEFAULT_TABLE table=default.Table_5e63f20b2f66459795a5289f47ffaf0e [id=10833938cf88466b834698420d7c01e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:58.998569 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9bf08ccf7054fedb9efcbb00354c71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.999972 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 97c6dce647064344b0b53a7ffaeafb75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.000267 1339 tablet_service.cc:1505] Processing CreateTablet for tablet b79801594e95434cbb8fe7b75b14602e (DEFAULT_TABLE table=default.Table_9291fbfe8e484be094f28ad94814b13f [id=7ec9391c52b1424eb52850f2c6ba4779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.000551 1480 tablet_service.cc:1552] Processing DeleteTablet for tablet a48249f1824a40b087b80a563816d485 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.000104 1482 tablet_service.cc:1505] Processing CreateTablet for tablet dc14b0cdb7f342ac956a7857ab3f35d7 (DEFAULT_TABLE table=default.Table_be61dd09c4c94f568d723c4bace3a038 [id=66be3fc672ea4a9e9fb5dde3ae5e5b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.001557 1768 tablet_bootstrap.cc:492] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:59.001667 1487 tablet_service.cc:1505] Processing CreateTablet for tablet fdfa5eea3b674a3389a156d1c72204df (DEFAULT_TABLE table=default.Table_d8ae96b95894421588b9b807f296a627 [id=96ea5a86f92b4d18b0d74f1092bd17b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.002007 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 635a132735704cfda69bf7c8deaea81d (DEFAULT_TABLE table=default.Table_4c3de3c56f624eee8d86a60154ec6f07 [id=93a61862707b4a7ba8a0ff349704dc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.002233 1768 ts_tablet_manager.cc:1403] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.107s user 0.004s sys 0.000s
I20251024 19:47:59.003574 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc14b0cdb7f342ac956a7857ab3f35d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.003589 1768 raft_consensus.cc:359] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.004125 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 (DEFAULT_TABLE table=default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:47:59.006703 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet a48249f1824a40b087b80a563816d485 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a48249f1824a40b087b80a563816d485 already in progress: creating tablet
I20251024 19:47:59.009843 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 635a132735704cfda69bf7c8deaea81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.014004 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdfa5eea3b674a3389a156d1c72204df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.017194 1941 raft_consensus.cc:493] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:59.017686 1941 raft_consensus.cc:515] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.019248 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97c6dce647064344b0b53a7ffaeafb75" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:47:59.019331 1941 leader_election.cc:290] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:47:59.021097 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3caecdff61ce48f7b7c3846bb380d21e" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:47:59.022199 1305 leader_election.cc:343] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:47:59.024895 1308 leader_election.cc:343] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:59.029796 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3caecdff61ce48f7b7c3846bb380d21e" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:47:59.030776 1308 leader_election.cc:343] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:59.031231 1308 leader_election.cc:304] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:59.035319 1941 raft_consensus.cc:2749] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:59.057853 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97c6dce647064344b0b53a7ffaeafb75" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:47:59.058162 1648 raft_consensus.cc:3060] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:59.062044 1882 leader_election.cc:290] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:59.069273 1648 raft_consensus.cc:2468] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:47:59.074393 1305 leader_election.cc:304] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
I20251024 19:47:59.083338 1882 raft_consensus.cc:2804] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:47:59.084807 1882 raft_consensus.cc:697] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:47:59.089031 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet a48249f1824a40b087b80a563816d485 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.089335 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7352d84226db467dad31ae5941842b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.093828 1768 raft_consensus.cc:385] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.093993 1768 raft_consensus.cc:740] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.094354 1768 consensus_queue.cc:260] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.095088 1768 ts_tablet_manager.cc:1434] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.093s user 0.002s sys 0.000s
I20251024 19:47:59.095645 1768 tablet_bootstrap.cc:492] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:59.100711 1768 tablet_bootstrap.cc:654] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.102172 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 187925082cdd4923a6e9148bf9f270fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.104681 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc14b0cdb7f342ac956a7857ab3f35d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.113106 1941 raft_consensus.cc:493] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:59.113322 1941 raft_consensus.cc:515] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.114195 1941 leader_election.cc:290] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:59.118461 1776 ts_tablet_manager.cc:1929] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.121546 1776 log.cc:1199] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/1745bbc68d6846fab1a6197f898754b4
I20251024 19:47:59.124925 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9bf08ccf7054fedb9efcbb00354c71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.131315 1776 ts_tablet_manager.cc:1950] T 1745bbc68d6846fab1a6197f898754b4 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:59.132251 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b79801594e95434cbb8fe7b75b14602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.141114 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdfa5eea3b674a3389a156d1c72204df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.146113 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 635a132735704cfda69bf7c8deaea81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.147275 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 1745bbc68d6846fab1a6197f898754b4 (table default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]) successfully deleted
W20251024 19:47:59.150053 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet c292ce055def4964ab143252c6b2b87e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c292ce055def4964ab143252c6b2b87e already in progress: creating tablet
I20251024 19:47:59.150602 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758df87acbdb4adaa83e1ecbc44e7cf1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:59.152742 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 030e690ce4df4818a6450ae293b0ffbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 030e690ce4df4818a6450ae293b0ffbd already in progress: creating tablet
I20251024 19:47:59.153008 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 59c7dbe9ff2c4869994c4f43d5676c30 (DEFAULT_TABLE table=default.Table_eb9c8019451d46fba31a87a0b6ac6ad8 [id=e5b1dd31c2114e8a95ba33981029255c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.153474 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c7dbe9ff2c4869994c4f43d5676c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:58.920996 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 87) took 1954 ms. Trace:
I20251024 19:47:59.153991 1774 rpcz_store.cc:276] 1024 19:47:56.954166 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:56.954225 (+ 59us) service_pool.cc:225] Handling call
1024 19:47:58.387517 (+1433292us) ts_tablet_manager.cc:1151] Deleting tablet 0397637f0778442a871528d5dd173689
1024 19:47:58.387529 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:58.808067 (+420538us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:58.903868 (+ 95801us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:58.908250 (+ 4382us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1433029}
I20251024 19:47:59.153895 1776 tablet_replica.cc:333] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:59.154676 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 0397637f0778442a871528d5dd173689 (table default.Table_59f512ddad6842eca4865d6387dee631 [id=e6d5a0f628404ad0bd19cc019911e269]) successfully deleted
I20251024 19:47:59.155258 1774 tablet_replica.cc:333] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:59.155545 1774 raft_consensus.cc:2243] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.155857 1774 raft_consensus.cc:2272] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.156597 1774 ts_tablet_manager.cc:1916] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.163729 1979 tablet.cc:1717] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e: Skipping requested alter to schema version 4, tablet already version 4
I20251024 19:47:59.166196 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f646bd05274cb6859c0b7f92b61352" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:59.166232 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f646bd05274cb6859c0b7f92b61352" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:59.166467 1648 raft_consensus.cc:2468] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:47:59.166872 1305 leader_election.cc:304] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:47:59.166988 1776 raft_consensus.cc:2243] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.167174 1776 raft_consensus.cc:2272] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.167245 1941 raft_consensus.cc:2804] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:47:59.167397 1941 raft_consensus.cc:493] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:47:59.167536 1941 raft_consensus.cc:3060] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:59.170979 1776 ts_tablet_manager.cc:1916] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.172194 1941 raft_consensus.cc:515] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.188469 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f646bd05274cb6859c0b7f92b61352" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:47:59.188717 1649 raft_consensus.cc:3060] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:59.193723 1649 raft_consensus.cc:2468] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:47:59.172839 1882 consensus_queue.cc:237] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.186947 1768 tablet_bootstrap.cc:492] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:59.194481 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65f646bd05274cb6859c0b7f92b61352" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:47:59.194588 1768 ts_tablet_manager.cc:1403] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.099s user 0.002s sys 0.003s
W20251024 19:47:59.196025 1308 leader_election.cc:343] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:59.198170 1941 leader_election.cc:290] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:59.200368 1305 leader_election.cc:304] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:47:59.205667 1980 raft_consensus.cc:2804] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
W20251024 19:47:59.205727 1308 leader_election.cc:343] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:59.205977 1980 raft_consensus.cc:697] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:47:59.204298 1768 raft_consensus.cc:359] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.206277 1768 raft_consensus.cc:385] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.206393 1768 raft_consensus.cc:740] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.206727 1980 consensus_queue.cc:237] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.207178 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_999546c6f492460bb6dc96ab55b7d3ad" } modify_external_catalogs: true
I20251024 19:47:59.207474 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_999546c6f492460bb6dc96ab55b7d3ad" } modify_external_catalogs: true
I20251024 19:47:59.213227 1185 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_a54ffafa18b848b79a290f58c9754bec [id=2fd5c541b06a46469e65ed0942d2c394] alter complete (version 4)
I20251024 19:47:59.214599 1768 consensus_queue.cc:260] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.216226 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_52c56313187f4435b2136108330e12b4" } new_table_name: "default.Table_96f7d0ed2ff04485945a33b8eda07570" modify_external_catalogs: true
I20251024 19:47:59.217501 1768 ts_tablet_manager.cc:1434] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.023s user 0.000s sys 0.002s
I20251024 19:47:59.217916 1768 tablet_bootstrap.cc:492] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:59.218031 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7f14fdc529ec48d68d03ca041f64a53f" } modify_external_catalogs: true
I20251024 19:47:59.218258 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7f14fdc529ec48d68d03ca041f64a53f" } modify_external_catalogs: true
I20251024 19:47:59.219111 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0e24823283b64679932dab322281252b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.219892 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e24823283b64679932dab322281252b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.222353 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4d53ff8bfef04c2f82bef077ab04d4a6" } new_table_name: "default.Table_a372d22e91854206b3523ecd087303ca" modify_external_catalogs: true
I20251024 19:47:59.224495 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5760bec3ecb94c9c94b3fd2a7ecdd4fd" } new_table_name: "default.Table_24200cd6f19d460cae963afb9b7f8724" modify_external_catalogs: true
I20251024 19:47:59.226756 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c10fcb6694c142e8b806f745a3ddc82e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.227597 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c10fcb6694c142e8b806f745a3ddc82e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.239077 1776 ts_tablet_manager.cc:1929] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.241591 1956 tablet.cc:1717] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532: Skipping requested alter to schema version 4, tablet already version 4
I20251024 19:47:59.253149 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_690da5d1c64e4889b9620a19138f55cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.254122 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_690da5d1c64e4889b9620a19138f55cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.247053 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 9b89a90bb4ca447fbaf34fe39001c261 (DEFAULT_TABLE table=default.Table_46251d3ce016440287b2a378eae2e9a7 [id=2fda7fc5a71646b79f5d848dca570135]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.257112 1776 log.cc:1199] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/047436af5f6d4f688bd83e33089d3ee4
I20251024 19:47:59.252983 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 65f646bd05274cb6859c0b7f92b61352 with delete_type TABLET_DATA_DELETED (replaced by tablet aebf76d4a66947ed9756643740418798) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.253196 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 0397637f0778442a871528d5dd173689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.253351 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.258211 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 9b89a90bb4ca447fbaf34fe39001c261 (DEFAULT_TABLE table=default.Table_46251d3ce016440287b2a378eae2e9a7 [id=2fda7fc5a71646b79f5d848dca570135]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.258343 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b89a90bb4ca447fbaf34fe39001c261. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.260083 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b89a90bb4ca447fbaf34fe39001c261. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.262975 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 9b89a90bb4ca447fbaf34fe39001c261 (DEFAULT_TABLE table=default.Table_46251d3ce016440287b2a378eae2e9a7 [id=2fda7fc5a71646b79f5d848dca570135]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.263573 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b89a90bb4ca447fbaf34fe39001c261. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.265416 1768 tablet_bootstrap.cc:654] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.265762 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 1745bbc68d6846fab1a6197f898754b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 31647a48fb834c80b6920c756a94250d) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.266040 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 047436af5f6d4f688bd83e33089d3ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.266284 1775 tablet_replica.cc:333] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:59.266528 1775 raft_consensus.cc:2243] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.266731 1775 raft_consensus.cc:2272] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.268014 1775 ts_tablet_manager.cc:1916] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.275650 1775 ts_tablet_manager.cc:1929] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.282099 1775 log.cc:1199] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/1745bbc68d6846fab1a6197f898754b4
W20251024 19:47:59.298388 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 037c8cc1e53447109fafc047c02f196f consensus state has no leader
W20251024 19:47:59.299371 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c357a2a81064b3891afda6d252898b9 consensus state has no leader
W20251024 19:47:59.299908 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7352d84226db467dad31ae5941842b80 consensus state has no leader
I20251024 19:47:59.305172 1776 ts_tablet_manager.cc:1950] T 047436af5f6d4f688bd83e33089d3ee4 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:59.306576 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 047436af5f6d4f688bd83e33089d3ee4 (table default.Table_812ae4fbc0fc4fdea1317e0d9794c50b [id=93f77b3367c04c9bb7e3e02cfa050c36]) successfully deleted
I20251024 19:47:59.306748 1776 tablet_replica.cc:333] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:59.306985 1776 raft_consensus.cc:2243] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.307224 1776 raft_consensus.cc:2272] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.307791 1776 ts_tablet_manager.cc:1916] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:47:59.313071 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 27b32126d04e480f8db7b2b20aba5873 has no consensus state
W20251024 19:47:59.313858 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c04082bd1b444b468c177bef239701c6 consensus state has no leader
I20251024 19:47:59.318188 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7dbe5740202642f1be406c4897a72541" } modify_external_catalogs: true
I20251024 19:47:59.318396 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7dbe5740202642f1be406c4897a72541" } modify_external_catalogs: true
I20251024 19:47:59.319043 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a0cee23cfd2c42a2960434cb38f6779e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.319635 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0cee23cfd2c42a2960434cb38f6779e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.321105 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_64aaab6eda0a443fbb49223835af43e3" } modify_external_catalogs: true
I20251024 19:47:59.321302 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_64aaab6eda0a443fbb49223835af43e3" } modify_external_catalogs: true
I20251024 19:47:59.321864 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_11634cbabbd6478484a0aed448ae9521"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.322366 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11634cbabbd6478484a0aed448ae9521 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.324055 1775 ts_tablet_manager.cc:1950] T 1745bbc68d6846fab1a6197f898754b4 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:59.324780 1767 tablet_bootstrap.cc:492] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:59.324522 1766 tablet_bootstrap.cc:492] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:59.325145 1766 ts_tablet_manager.cc:1403] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.471s user 0.009s sys 0.020s
I20251024 19:47:59.326218 1766 raft_consensus.cc:359] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.326406 1766 raft_consensus.cc:385] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.326516 1766 raft_consensus.cc:740] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.326754 1766 consensus_queue.cc:260] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.326998 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9110872a9b40450bb070d0c1db2d2b84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:59.327277 1766 ts_tablet_manager.cc:1434] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251024 19:47:59.327509 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9110872a9b40450bb070d0c1db2d2b84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.327839 1766 tablet_bootstrap.cc:492] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:59.329699 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_51982d999b0847ec87fc62a473782a5c" } modify_external_catalogs: true
I20251024 19:47:59.329867 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_51982d999b0847ec87fc62a473782a5c" } modify_external_catalogs: true
I20251024 19:47:59.330318 1766 tablet_bootstrap.cc:654] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.330446 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ade7993306ee438baebbf639e2109768"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.330958 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ade7993306ee438baebbf639e2109768 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.334559 1768 tablet_bootstrap.cc:492] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:59.334759 1768 ts_tablet_manager.cc:1403] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.117s user 0.002s sys 0.002s
I20251024 19:47:59.335309 1774 ts_tablet_manager.cc:1929] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.335434 1774 log.cc:1199] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/fc8c01a4c07a42a0bc48a58313090e68
I20251024 19:47:59.339079 1768 raft_consensus.cc:359] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.339363 1768 raft_consensus.cc:385] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.339391 1774 ts_tablet_manager.cc:1950] T fc8c01a4c07a42a0bc48a58313090e68 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:59.339453 1768 raft_consensus.cc:740] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.339771 1768 consensus_queue.cc:260] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.340454 1768 ts_tablet_manager.cc:1434] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251024 19:47:59.340989 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 90) took 2251 ms. Trace:
I20251024 19:47:59.341097 1774 rpcz_store.cc:276] 1024 19:47:57.089803 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.100730 (+ 10927us) service_pool.cc:225] Handling call
1024 19:47:59.155061 (+2054331us) ts_tablet_manager.cc:1151] Deleting tablet fc8c01a4c07a42a0bc48a58313090e68
1024 19:47:59.155072 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:59.276821 (+121749us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.335164 (+ 58343us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.340938 (+ 5774us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":522,"tablet-delete.queue_time_us":2042605}
I20251024 19:47:59.341604 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet fc8c01a4c07a42a0bc48a58313090e68 (table default.Table_401da5a2f0dc42bebcb8f9076d40a8b8 [id=d955ab5050fa4039b48b0c9235b687cd]) successfully deleted
I20251024 19:47:59.342010 1768 tablet_bootstrap.cc:492] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:59.344242 1768 tablet_bootstrap.cc:654] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.345664 1774 tablet_replica.cc:333] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:59.345872 1774 raft_consensus.cc:2243] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.346053 1774 raft_consensus.cc:2272] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.348809 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 1745bbc68d6846fab1a6197f898754b4 (table default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]) successfully deleted
I20251024 19:47:59.349411 1775 tablet_replica.cc:333] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:59.350160 1775 raft_consensus.cc:2243] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.350665 1775 raft_consensus.cc:2272] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.351826 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet b9b1e0338fd14634a1e921735ae61289 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:59.354738 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 03cdbd1b62b741249ad78562a7edfd22 (DEFAULT_TABLE table=default.Table_31d09f80ce0a4953b4764b8425875f89 [id=ca742ae585e0406abf43193a23aa9731]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.355278 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03cdbd1b62b741249ad78562a7edfd22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.338762 1776 ts_tablet_manager.cc:1929] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.360788 1776 log.cc:1199] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/97c6dce647064344b0b53a7ffaeafb75
I20251024 19:47:59.360981 1767 ts_tablet_manager.cc:1403] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.379s user 0.031s sys 0.150s
I20251024 19:47:59.361197 1776 ts_tablet_manager.cc:1950] T 97c6dce647064344b0b53a7ffaeafb75 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:59.362188 1767 raft_consensus.cc:359] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:59.362448 1776 tablet_replica.cc:333] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:59.362646 1776 raft_consensus.cc:2243] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.362794 1776 raft_consensus.cc:2272] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.359354 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 03cdbd1b62b741249ad78562a7edfd22 (DEFAULT_TABLE table=default.Table_31d09f80ce0a4953b4764b8425875f89 [id=ca742ae585e0406abf43193a23aa9731]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.363464 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03cdbd1b62b741249ad78562a7edfd22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.363586 1776 ts_tablet_manager.cc:1916] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:47:59.364180 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 43667.5us
I20251024 19:47:59.365690 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 97c6dce647064344b0b53a7ffaeafb75 (table default.Table_85e6c4c107c94e7abfc7c3396aaf6e81 [id=1b078c2814dd4ff99c1e2ee01d0159cf]) successfully deleted
W20251024 19:47:59.366194 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet a48249f1824a40b087b80a563816d485 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a48249f1824a40b087b80a563816d485 already in progress: creating tablet
I20251024 19:47:59.367627 1774 ts_tablet_manager.cc:1916] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.367720 1624 tablet_service.cc:1505] Processing CreateTablet for tablet b88a6a5237d146f5b6f082ca3af25b2b (DEFAULT_TABLE table=default.Table_6eb10ae51f3a43c2ab15f362708fba94 [id=f3080df28ee94876bb6b790cea45a4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.368112 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88a6a5237d146f5b6f082ca3af25b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.369961 1340 tablet_service.cc:1505] Processing CreateTablet for tablet b88a6a5237d146f5b6f082ca3af25b2b (DEFAULT_TABLE table=default.Table_6eb10ae51f3a43c2ab15f362708fba94 [id=f3080df28ee94876bb6b790cea45a4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.370329 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 27b32126d04e480f8db7b2b20aba5873 (DEFAULT_TABLE table=default.Table_a372d22e91854206b3523ecd087303ca [id=dfffb9501a0549198d17b197ef05871e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.370450 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88a6a5237d146f5b6f082ca3af25b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.370767 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b32126d04e480f8db7b2b20aba5873. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.373978 1626 tablet_service.cc:1505] Processing CreateTablet for tablet 27b32126d04e480f8db7b2b20aba5873 (DEFAULT_TABLE table=default.Table_a372d22e91854206b3523ecd087303ca [id=dfffb9501a0549198d17b197ef05871e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.374388 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b32126d04e480f8db7b2b20aba5873. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.374735 1776 ts_tablet_manager.cc:1929] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.374872 1776 log.cc:1199] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/65f646bd05274cb6859c0b7f92b61352
I20251024 19:47:59.375267 1776 ts_tablet_manager.cc:1950] T 65f646bd05274cb6859c0b7f92b61352 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:59.375896 1775 ts_tablet_manager.cc:1916] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.376269 1776 tablet_replica.cc:333] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:59.376458 1776 raft_consensus.cc:2243] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.355162 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 03cdbd1b62b741249ad78562a7edfd22 (DEFAULT_TABLE table=default.Table_31d09f80ce0a4953b4764b8425875f89 [id=ca742ae585e0406abf43193a23aa9731]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.366470 1484 tablet_service.cc:1505] Processing CreateTablet for tablet b88a6a5237d146f5b6f082ca3af25b2b (DEFAULT_TABLE table=default.Table_6eb10ae51f3a43c2ab15f362708fba94 [id=f3080df28ee94876bb6b790cea45a4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.376645 1776 raft_consensus.cc:2272] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.376974 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88a6a5237d146f5b6f082ca3af25b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.377477 1776 ts_tablet_manager.cc:1916] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.368795 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 27b32126d04e480f8db7b2b20aba5873 (DEFAULT_TABLE table=default.Table_a372d22e91854206b3523ecd087303ca [id=dfffb9501a0549198d17b197ef05871e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.373677 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 701c93a600964c20a275f5942a2ff863 (DEFAULT_TABLE table=default.Table_7dc53b918ca7443b807133df217c0f75 [id=9ce6ac48924e45e3a482f71740edb4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.378049 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 65f646bd05274cb6859c0b7f92b61352 (table default.Table_d09f149249c44c50adcb71321c133f2d [id=665389c7bde54b7786edabcb91a9b33b]) successfully deleted
I20251024 19:47:59.362454 1767 raft_consensus.cc:385] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.378530 1767 raft_consensus.cc:740] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.379279 1767 consensus_queue.cc:260] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:59.375267 1766 tablet_bootstrap.cc:492] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:59.382522 1188 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]). Expected version 3 got 0
I20251024 19:47:59.383246 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet 7c357a2a81064b3891afda6d252898b9 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:59.385648 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b32126d04e480f8db7b2b20aba5873. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.385825 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet 97c6dce647064344b0b53a7ffaeafb75 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:59.386327 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet 73ffe342f541447d8519ee24bc98a0f3 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:59.388737 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 5368789df6024d7daa9ccc2257137d58 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:59.390615 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 701c93a600964c20a275f5942a2ff863. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.394846 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet e2d97caee586418a8981108465f6b704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.404315 1776 ts_tablet_manager.cc:1929] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.404525 1776 log.cc:1199] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/6f21ffa44d7f48d08ed17ebf6c8ff29c
I20251024 19:47:59.404958 1776 ts_tablet_manager.cc:1950] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:59.406281 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c (table default.Table_656be68a26b545d2b254abc2edb15254 [id=a1c4569aa6934d4a99787cd9e3240edb]) successfully deleted
I20251024 19:47:59.406634 1776 tablet_replica.cc:333] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:59.406947 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03cdbd1b62b741249ad78562a7edfd22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.407856 1766 ts_tablet_manager.cc:1403] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.002s
I20251024 19:47:59.412081 1776 raft_consensus.cc:2243] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.412305 1776 raft_consensus.cc:2272] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.413132 1776 ts_tablet_manager.cc:1916] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.414218 1626 tablet_service.cc:1505] Processing CreateTablet for tablet 701c93a600964c20a275f5942a2ff863 (DEFAULT_TABLE table=default.Table_7dc53b918ca7443b807133df217c0f75 [id=9ce6ac48924e45e3a482f71740edb4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.414798 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 701c93a600964c20a275f5942a2ff863. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.418684 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 5fd7726c532145c6b44ea9a37544510d
I20251024 19:47:59.419800 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fd7726c532145c6b44ea9a37544510d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.421728 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fd7726c532145c6b44ea9a37544510d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.423348 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9623d7baf0174a019e2353a5effef081
I20251024 19:47:59.424199 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b6661b9e9a174849b6093d09cd8c1251
I20251024 19:47:59.424907 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 73ffe342f541447d8519ee24bc98a0f3
I20251024 19:47:59.425438 1186 catalog_manager.cc:6719] ReplaceTablet: tablet 73ffe342f541447d8519ee24bc98a0f3 deleted and replaced by tablet 505e4fb0755a4caaa462e5f5154bfee2
I20251024 19:47:59.425832 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ca0527af718f494a8bbcc66e7b325bed
I20251024 19:47:59.426445 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9fffc42ddda14bd28800917ff2f0ced5
I20251024 19:47:59.426857 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet ca0527af718f494a8bbcc66e7b325bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.426057 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 73ffe342f541447d8519ee24bc98a0f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 505e4fb0755a4caaa462e5f5154bfee2) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.427045 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7c357a2a81064b3891afda6d252898b9
I20251024 19:47:59.427495 1188 catalog_manager.cc:6719] ReplaceTablet: tablet 7c357a2a81064b3891afda6d252898b9 deleted and replaced by tablet d38154d3ffd34f4b8d7305b1c23a4fae
I20251024 19:47:59.427923 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c357a2a81064b3891afda6d252898b9 with delete_type TABLET_DATA_DELETED (replaced by tablet d38154d3ffd34f4b8d7305b1c23a4fae) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.428692 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet b6661b9e9a174849b6093d09cd8c1251 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.428925 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fffc42ddda14bd28800917ff2f0ced5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.429102 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 9623d7baf0174a019e2353a5effef081 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.429028 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_31fb0d4536b94892aeceb91307c74837"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.429616 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31fb0d4536b94892aeceb91307c74837 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.431905 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet ca0527af718f494a8bbcc66e7b325bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.431986 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 73ffe342f541447d8519ee24bc98a0f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 505e4fb0755a4caaa462e5f5154bfee2) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.432165 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fffc42ddda14bd28800917ff2f0ced5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.432391 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c357a2a81064b3891afda6d252898b9 with delete_type TABLET_DATA_DELETED (replaced by tablet d38154d3ffd34f4b8d7305b1c23a4fae) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.432428 1774 ts_tablet_manager.cc:1929] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.432581 1774 log.cc:1199] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/15c08dcf379f4df9a0e461d6dcfeeac4
I20251024 19:47:59.433030 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet b6661b9e9a174849b6093d09cd8c1251 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.437537 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_231e4d38da7a4fe4b4e6b33e854aa1d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.438218 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_231e4d38da7a4fe4b4e6b33e854aa1d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.441386 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 9623d7baf0174a019e2353a5effef081 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.442993 1767 ts_tablet_manager.cc:1434] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.082s user 0.003s sys 0.000s
I20251024 19:47:59.443405 1767 tablet_bootstrap.cc:492] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:59.441635 1766 raft_consensus.cc:359] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.446439 1766 raft_consensus.cc:385] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.446540 1766 raft_consensus.cc:740] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.447517 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d1b030ba4f784a2b80386b386edec4d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.448397 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1b030ba4f784a2b80386b386edec4d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.449681 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 701c93a600964c20a275f5942a2ff863 (DEFAULT_TABLE table=default.Table_7dc53b918ca7443b807133df217c0f75 [id=9ce6ac48924e45e3a482f71740edb4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.450150 1184 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]). Expected version 3 got 0
I20251024 19:47:59.453758 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet ca0527af718f494a8bbcc66e7b325bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.453851 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fffc42ddda14bd28800917ff2f0ced5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.454041 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c357a2a81064b3891afda6d252898b9 with delete_type TABLET_DATA_DELETED (replaced by tablet d38154d3ffd34f4b8d7305b1c23a4fae) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.454078 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fd7726c532145c6b44ea9a37544510d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.454236 1776 ts_tablet_manager.cc:1929] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.454393 1776 log.cc:1199] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/0397637f0778442a871528d5dd173689
I20251024 19:47:59.454957 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 701c93a600964c20a275f5942a2ff863. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.455322 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 9623d7baf0174a019e2353a5effef081 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.455331 1766 consensus_queue.cc:260] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.455519 1776 ts_tablet_manager.cc:1950] T 0397637f0778442a871528d5dd173689 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:59.455991 1766 ts_tablet_manager.cc:1434] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.048s user 0.000s sys 0.002s
I20251024 19:47:59.456604 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 73ffe342f541447d8519ee24bc98a0f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 505e4fb0755a4caaa462e5f5154bfee2) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.456687 1766 tablet_bootstrap.cc:492] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:59.457125 1768 tablet_bootstrap.cc:492] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:59.457281 1768 ts_tablet_manager.cc:1403] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.115s user 0.012s sys 0.017s
I20251024 19:47:59.457974 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet b6661b9e9a174849b6093d09cd8c1251 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.461074 1768 raft_consensus.cc:359] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.461266 1768 raft_consensus.cc:385] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.461802 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 0397637f0778442a871528d5dd173689 (table default.Table_59f512ddad6842eca4865d6387dee631 [id=e6d5a0f628404ad0bd19cc019911e269]) successfully deleted
I20251024 19:47:59.461815 1768 raft_consensus.cc:740] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.462078 1768 consensus_queue.cc:260] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.462767 1768 ts_tablet_manager.cc:1434] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:47:59.463086 1768 tablet_bootstrap.cc:492] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:47:59.463655 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 5fd7726c532145c6b44ea9a37544510d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fd7726c532145c6b44ea9a37544510d already in progress: creating tablet
I20251024 19:47:59.464426 1774 ts_tablet_manager.cc:1950] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:47:59.464532 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet ca0527af718f494a8bbcc66e7b325bed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca0527af718f494a8bbcc66e7b325bed already in progress: creating tablet
W20251024 19:47:59.465551 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 73ffe342f541447d8519ee24bc98a0f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73ffe342f541447d8519ee24bc98a0f3 already in progress: creating tablet
W20251024 19:47:59.466378 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9fffc42ddda14bd28800917ff2f0ced5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fffc42ddda14bd28800917ff2f0ced5 already in progress: creating tablet
I20251024 19:47:59.467195 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 96) took 2376 ms. Trace:
W20251024 19:47:59.467202 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7c357a2a81064b3891afda6d252898b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c357a2a81064b3891afda6d252898b9 already in progress: creating tablet
I20251024 19:47:59.467295 1774 rpcz_store.cc:276] 1024 19:47:57.090436 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.100995 (+ 10559us) service_pool.cc:225] Handling call
1024 19:47:59.345597 (+2244602us) ts_tablet_manager.cc:1151] Deleting tablet 15c08dcf379f4df9a0e461d6dcfeeac4
1024 19:47:59.345606 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:59.423858 (+ 78252us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.432303 (+ 8445us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.467124 (+ 34821us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2230973}
I20251024 19:47:59.467656 1774 tablet_replica.cc:333] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:59.467852 1774 raft_consensus.cc:2243] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.467909 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 15c08dcf379f4df9a0e461d6dcfeeac4 (table default.Table_96c033f8e7ab4c919b38c80fabe3e367 [id=473fc6762db94b959bb9a260b11140f5]) successfully deleted
I20251024 19:47:59.468032 1774 raft_consensus.cc:2272] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:47:59.468057 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b6661b9e9a174849b6093d09cd8c1251 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6661b9e9a174849b6093d09cd8c1251 already in progress: creating tablet
I20251024 19:47:59.468561 1774 ts_tablet_manager.cc:1916] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:47:59.469273 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9623d7baf0174a019e2353a5effef081 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9623d7baf0174a019e2353a5effef081 already in progress: creating tablet
I20251024 19:47:59.471877 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 97c6dce647064344b0b53a7ffaeafb75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.475478 1767 tablet_bootstrap.cc:654] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.477610 1768 tablet_bootstrap.cc:654] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.483517 1766 tablet_bootstrap.cc:654] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.484086 1774 ts_tablet_manager.cc:1929] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.484280 1774 log.cc:1199] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/ae86e74f810c4b989ac633f59e6ac445
I20251024 19:47:59.484762 1774 ts_tablet_manager.cc:1950] T ae86e74f810c4b989ac633f59e6ac445 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:59.486974 1775 ts_tablet_manager.cc:1929] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.490988 1775 log.cc:1199] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/047436af5f6d4f688bd83e33089d3ee4
W20251024 19:47:59.490979 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 7c357a2a81064b3891afda6d252898b9 (table default.Table_b15672d0c13d4c939f61e30a2feb96e6 [id=ed879934f5754b92b8cff234f3f66333], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7c357a2a81064b3891afda6d252898b9 consensus state has no leader
I20251024 19:47:59.491883 1775 ts_tablet_manager.cc:1950] T 047436af5f6d4f688bd83e33089d3ee4 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:59.493177 1775 tablet_replica.cc:333] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:59.493333 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 047436af5f6d4f688bd83e33089d3ee4 (table default.Table_812ae4fbc0fc4fdea1317e0d9794c50b [id=93f77b3367c04c9bb7e3e02cfa050c36]) successfully deleted
I20251024 19:47:59.498567 1775 raft_consensus.cc:2243] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.498977 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 95) took 2408 ms. Trace:
I20251024 19:47:59.500952 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0e24823283b64679932dab322281252b" } new_table_name: "default.Table_bedd8edc24014953ac8c7dcecf18f128" modify_external_catalogs: true
I20251024 19:47:59.501418 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a0cee23cfd2c42a2960434cb38f6779e" } new_table_name: "default.Table_d9b6a6d63d7b424991e461c8b6065823" modify_external_catalogs: true
I20251024 19:47:59.501356 1774 rpcz_store.cc:276] 1024 19:47:57.090347 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.100970 (+ 10623us) service_pool.cc:225] Handling call
1024 19:47:59.467554 (+2366584us) ts_tablet_manager.cc:1151] Deleting tablet ae86e74f810c4b989ac633f59e6ac445
1024 19:47:59.467564 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:59.471908 (+ 4344us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.483859 (+ 11951us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.498925 (+ 15066us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2349288}
I20251024 19:47:59.501971 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet ae86e74f810c4b989ac633f59e6ac445 (table default.Table_ea60f0e4363c406d9ccff5f16b8ecce9 [id=071606963978424489f30719fa42baa3]) successfully deleted
I20251024 19:47:59.503255 1768 tablet_bootstrap.cc:492] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:59.503573 1768 ts_tablet_manager.cc:1403] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.007s
I20251024 19:47:59.504534 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5a1d470fc7bc493393e47da7db62dfca" } new_table_name: "default.Table_894d6739340148a995dd849a205c19c3" modify_external_catalogs: true
I20251024 19:47:59.504992 1768 raft_consensus.cc:359] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.505209 1768 raft_consensus.cc:385] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.505290 1768 raft_consensus.cc:740] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.505662 1768 consensus_queue.cc:260] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.508742 1768 ts_tablet_manager.cc:1434] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251024 19:47:59.509220 1768 tablet_bootstrap.cc:492] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:59.514351 1768 tablet_bootstrap.cc:654] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.516541 1775 raft_consensus.cc:2272] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.517791 1774 tablet_replica.cc:333] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:59.518141 1774 raft_consensus.cc:2243] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.518424 1774 raft_consensus.cc:2272] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.519258 1774 ts_tablet_manager.cc:1916] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.519423 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_573ec53c3d2c4de8864f399b291d8aa9" } new_table_name: "default.Table_f27b327a6f134fcbb33611d8bfcfc697" modify_external_catalogs: true
I20251024 19:47:59.520933 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_97c6fc690615425aa5af0d8c0dd249af" } new_table_name: "default.Table_cde3670bf0d04dc0b2049193bcee10d4" modify_external_catalogs: true
I20251024 19:47:59.522220 1775 ts_tablet_manager.cc:1916] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.522408 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a1d09325af844d71b290d95b8a4aaf12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.522967 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1d09325af844d71b290d95b8a4aaf12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.524174 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e378095ddecc4dcf801a1568160c34b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.524720 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e378095ddecc4dcf801a1568160c34b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.526594 1626 tablet_service.cc:1505] Processing CreateTablet for tablet cef80e4d14bb4e719bceebf43041d079 (DEFAULT_TABLE table=default.Table_ade7993306ee438baebbf639e2109768 [id=6318d0634b77498a8c1ff8618b8ccf4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.527102 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef80e4d14bb4e719bceebf43041d079. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.527161 1338 tablet_service.cc:1505] Processing CreateTablet for tablet cef80e4d14bb4e719bceebf43041d079 (DEFAULT_TABLE table=default.Table_ade7993306ee438baebbf639e2109768 [id=6318d0634b77498a8c1ff8618b8ccf4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.527187 1486 tablet_service.cc:1505] Processing CreateTablet for tablet cef80e4d14bb4e719bceebf43041d079 (DEFAULT_TABLE table=default.Table_ade7993306ee438baebbf639e2109768 [id=6318d0634b77498a8c1ff8618b8ccf4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.527568 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef80e4d14bb4e719bceebf43041d079. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.527583 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef80e4d14bb4e719bceebf43041d079. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.534574 1767 tablet_bootstrap.cc:492] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:59.534945 1767 ts_tablet_manager.cc:1403] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.091s user 0.016s sys 0.005s
W20251024 19:47:59.535352 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07e75c7f35544bec82cb994dea5ee4ef consensus state has no leader
W20251024 19:47:59.535514 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ffd446c269448d38cc8d987a53aecf9 consensus state has no leader
I20251024 19:47:59.536054 1767 raft_consensus.cc:359] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.536567 1767 raft_consensus.cc:385] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.536737 1767 raft_consensus.cc:740] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.538609 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 607ae453221d422e9d07882830c02275 (DEFAULT_TABLE table=default.Table_9110872a9b40450bb070d0c1db2d2b84 [id=2656b4a933364e6a98f1ee5f91f0b2ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.539036 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607ae453221d422e9d07882830c02275. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.539079 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 607ae453221d422e9d07882830c02275 (DEFAULT_TABLE table=default.Table_9110872a9b40450bb070d0c1db2d2b84 [id=2656b4a933364e6a98f1ee5f91f0b2ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.539496 1766 tablet_bootstrap.cc:492] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:59.539716 1766 ts_tablet_manager.cc:1403] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.083s user 0.000s sys 0.015s
I20251024 19:47:59.540273 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 607ae453221d422e9d07882830c02275 (DEFAULT_TABLE table=default.Table_9110872a9b40450bb070d0c1db2d2b84 [id=2656b4a933364e6a98f1ee5f91f0b2ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.540848 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607ae453221d422e9d07882830c02275. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:59.541886 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a12e415590a4c71aab4ad7d30412c83 consensus state has no leader
I20251024 19:47:59.545002 1774 ts_tablet_manager.cc:1929] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.545146 1774 log.cc:1199] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/95d7fbfa8ae54da49dd5603294faa7d3
I20251024 19:47:59.545500 1774 ts_tablet_manager.cc:1950] T 95d7fbfa8ae54da49dd5603294faa7d3 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:59.546336 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 110) took 1835 ms. Trace:
I20251024 19:47:59.546484 1774 rpcz_store.cc:276] 1024 19:47:57.711146 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.711222 (+ 76us) service_pool.cc:225] Handling call
1024 19:47:59.517704 (+1806482us) ts_tablet_manager.cc:1151] Deleting tablet 95d7fbfa8ae54da49dd5603294faa7d3
1024 19:47:59.517718 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:59.541616 (+ 23898us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.544895 (+ 3279us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.546274 (+ 1379us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806137}
I20251024 19:47:59.547044 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 95d7fbfa8ae54da49dd5603294faa7d3 (table default.Table_52ec55c23c394294a68850ab6b33aabe [id=fb0e7cab20f44aaebdb09862bd193f41]) successfully deleted
I20251024 19:47:59.548211 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 4243f11bf2b34b30b10766c19e2c6409 (DEFAULT_TABLE table=default.Table_11634cbabbd6478484a0aed448ae9521 [id=d3b067d90c5943449b53edca7c6eb475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.548600 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4243f11bf2b34b30b10766c19e2c6409. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:59.548782 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 452f77091adc4cf09de2231f2f6d845b consensus state has no leader
I20251024 19:47:59.551162 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 4243f11bf2b34b30b10766c19e2c6409 (DEFAULT_TABLE table=default.Table_11634cbabbd6478484a0aed448ae9521 [id=d3b067d90c5943449b53edca7c6eb475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.551319 1767 consensus_queue.cc:260] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:47:59.551651 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad055dc3387b414bb55d09b9683ba17e consensus state has no leader
I20251024 19:47:59.551826 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4243f11bf2b34b30b10766c19e2c6409. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.551328 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607ae453221d422e9d07882830c02275. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.553100 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 4243f11bf2b34b30b10766c19e2c6409 (DEFAULT_TABLE table=default.Table_11634cbabbd6478484a0aed448ae9521 [id=d3b067d90c5943449b53edca7c6eb475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.553141 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 4ffd446c269448d38cc8d987a53aecf9 (DEFAULT_TABLE table=default.Table_bedd8edc24014953ac8c7dcecf18f128 [id=2ae1c808837e48df97c4efb01a684da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.553619 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4243f11bf2b34b30b10766c19e2c6409. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.553752 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 07e75c7f35544bec82cb994dea5ee4ef (DEFAULT_TABLE table=default.Table_d9b6a6d63d7b424991e461c8b6065823 [id=450dc07c78d343aea5c8c2d2a6e26444]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.555079 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07e75c7f35544bec82cb994dea5ee4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.556490 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ffd446c269448d38cc8d987a53aecf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.551980 1767 ts_tablet_manager.cc:1434] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.017s user 0.001s sys 0.002s
I20251024 19:47:59.557515 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_775e17f58fef4f2b850035a03e411a08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:59.557902 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_efcd2af1913c4a73ba5148272aa880aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.558290 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_775e17f58fef4f2b850035a03e411a08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:59.558434 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efcd2af1913c4a73ba5148272aa880aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.559001 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 07e75c7f35544bec82cb994dea5ee4ef (DEFAULT_TABLE table=default.Table_d9b6a6d63d7b424991e461c8b6065823 [id=450dc07c78d343aea5c8c2d2a6e26444]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.559442 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07e75c7f35544bec82cb994dea5ee4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.559630 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a86884fdc4364764911f8b802fc35102"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.560076 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a86884fdc4364764911f8b802fc35102 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.563267 1767 tablet_bootstrap.cc:492] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:59.569133 1882 raft_consensus.cc:493] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:59.569295 1882 raft_consensus.cc:515] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.569986 1882 leader_election.cc:290] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:59.570785 1774 tablet_replica.cc:333] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:59.571180 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 4ffd446c269448d38cc8d987a53aecf9 (DEFAULT_TABLE table=default.Table_bedd8edc24014953ac8c7dcecf18f128 [id=2ae1c808837e48df97c4efb01a684da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.573235 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ffd446c269448d38cc8d987a53aecf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.578531 1342 tablet_service.cc:1505] Processing CreateTablet for tablet ce8c179e98944df0b510a0e8daf3920f (DEFAULT_TABLE table=default.Table_690da5d1c64e4889b9620a19138f55cc [id=870c629198ed4f53b9317b1340e75ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.578751 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 230606a0ce084b7bab0e41e48cf8fa89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.578938 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce8c179e98944df0b510a0e8daf3920f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.579346 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "230606a0ce084b7bab0e41e48cf8fa89" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:59.581000 1767 tablet_bootstrap.cc:654] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.581039 1776 tablet_replica.cc:333] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:59.581300 1623 tablet_service.cc:1505] Processing CreateTablet for tablet ce8c179e98944df0b510a0e8daf3920f (DEFAULT_TABLE table=default.Table_690da5d1c64e4889b9620a19138f55cc [id=870c629198ed4f53b9317b1340e75ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.581471 1617 tablet_service.cc:1505] Processing CreateTablet for tablet e03c488d0f264684b0897a81a04d7466 (DEFAULT_TABLE table=default.Table_c10fcb6694c142e8b806f745a3ddc82e [id=5c054cd9fe2c4684854e0da6e68bc7d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.582229 1340 tablet_service.cc:1505] Processing CreateTablet for tablet e03c488d0f264684b0897a81a04d7466 (DEFAULT_TABLE table=default.Table_c10fcb6694c142e8b806f745a3ddc82e [id=5c054cd9fe2c4684854e0da6e68bc7d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.582968 1776 raft_consensus.cc:2243] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.583549 1776 raft_consensus.cc:2272] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:47:59.586040 1305 leader_election.cc:343] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:59.587103 1776 ts_tablet_manager.cc:1916] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.587781 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e03c488d0f264684b0897a81a04d7466. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.588280 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce8c179e98944df0b510a0e8daf3920f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.604215 1189 master_service.cc:954] ReplaceTablet: received request to replace tablet 230606a0ce084b7bab0e41e48cf8fa89 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:59.604594 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1c89d244042a4852a9cb4113f6726332"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:59.604230 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0d796a82cf4b4e0f9cba35051c69293d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.605546 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d796a82cf4b4e0f9cba35051c69293d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:59.610667 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c89d244042a4852a9cb4113f6726332 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.623862 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5f1d0b12b2034f5185f21c87157cf051" } modify_external_catalogs: true
I20251024 19:47:59.624090 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5f1d0b12b2034f5185f21c87157cf051" } modify_external_catalogs: true
I20251024 19:47:59.625087 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet 2037fcca87b94b5fb31639b4d07307b0 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:59.625478 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a372d22e91854206b3523ecd087303ca" } modify_external_catalogs: true
I20251024 19:47:59.625711 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a372d22e91854206b3523ecd087303ca" } modify_external_catalogs: true
I20251024 19:47:59.627539 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d09f149249c44c50adcb71321c133f2d" } modify_external_catalogs: true
I20251024 19:47:59.627692 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d0ae44b1936f452d9c75bd4b888bfb30" } modify_external_catalogs: true
I20251024 19:47:59.627873 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d0ae44b1936f452d9c75bd4b888bfb30" } modify_external_catalogs: true
I20251024 19:47:59.627887 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d09f149249c44c50adcb71321c133f2d" } modify_external_catalogs: true
I20251024 19:47:59.628255 1774 raft_consensus.cc:2243] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.628420 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a54ffafa18b848b79a290f58c9754bec" } modify_external_catalogs: true
I20251024 19:47:59.628499 1774 raft_consensus.cc:2272] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.628603 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a54ffafa18b848b79a290f58c9754bec" } modify_external_catalogs: true
I20251024 19:47:59.629812 1767 tablet_bootstrap.cc:492] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:59.629910 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e03c488d0f264684b0897a81a04d7466. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.629997 1767 ts_tablet_manager.cc:1403] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.020s
I20251024 19:47:59.632997 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 07e75c7f35544bec82cb994dea5ee4ef (DEFAULT_TABLE table=default.Table_d9b6a6d63d7b424991e461c8b6065823 [id=450dc07c78d343aea5c8c2d2a6e26444]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.634783 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet fca73ab61f0c47df9186459caf2b9a32 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:59.638497 1482 tablet_service.cc:1505] Processing CreateTablet for tablet ce8c179e98944df0b510a0e8daf3920f (DEFAULT_TABLE table=default.Table_690da5d1c64e4889b9620a19138f55cc [id=870c629198ed4f53b9317b1340e75ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.639708 1768 tablet_bootstrap.cc:492] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:59.639873 1768 ts_tablet_manager.cc:1403] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.131s user 0.004s sys 0.002s
I20251024 19:47:59.640754 1768 raft_consensus.cc:359] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:59.640955 1768 raft_consensus.cc:385] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.641359 1768 raft_consensus.cc:740] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.641644 1768 consensus_queue.cc:260] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:47:59.642010 1308 consensus_peers.cc:597] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 047436af5f6d4f688bd83e33089d3ee4. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:47:59.642488 1768 ts_tablet_manager.cc:1434] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:47:59.642849 1768 tablet_bootstrap.cc:492] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:59.645082 1768 tablet_bootstrap.cc:654] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.648989 1774 ts_tablet_manager.cc:1916] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:47:59.653292 1305 consensus_peers.cc:597] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 047436af5f6d4f688bd83e33089d3ee4. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:47:59.654084 1767 raft_consensus.cc:359] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.654405 1767 raft_consensus.cc:385] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.656736 1767 raft_consensus.cc:740] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.657016 1767 consensus_queue.cc:260] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.658705 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6f376570019346199a3c8c77d879fa1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.660048 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f376570019346199a3c8c77d879fa1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.662124 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 4ffd446c269448d38cc8d987a53aecf9 (DEFAULT_TABLE table=default.Table_bedd8edc24014953ac8c7dcecf18f128 [id=2ae1c808837e48df97c4efb01a684da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.662562 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "230606a0ce084b7bab0e41e48cf8fa89" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:47:59.663069 1488 tablet_service.cc:1505] Processing CreateTablet for tablet e03c488d0f264684b0897a81a04d7466 (DEFAULT_TABLE table=default.Table_c10fcb6694c142e8b806f745a3ddc82e [id=5c054cd9fe2c4684854e0da6e68bc7d8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:47:59.663115 1308 leader_election.cc:343] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:59.663232 1308 leader_election.cc:304] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:59.663479 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce8c179e98944df0b510a0e8daf3920f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.663694 1767 ts_tablet_manager.cc:1434] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20251024 19:47:59.664038 1767 tablet_bootstrap.cc:492] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:47:59.669857 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ffd446c269448d38cc8d987a53aecf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.673792 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e03c488d0f264684b0897a81a04d7466. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.681586 1882 raft_consensus.cc:2749] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:59.682034 1774 ts_tablet_manager.cc:1929] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.682168 1774 log.cc:1199] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/6f21ffa44d7f48d08ed17ebf6c8ff29c
I20251024 19:47:59.682602 1774 ts_tablet_manager.cc:1950] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:59.682642 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07e75c7f35544bec82cb994dea5ee4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.683462 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 114) took 1971 ms. Trace:
I20251024 19:47:59.683570 1774 rpcz_store.cc:276] 1024 19:47:57.711645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.711711 (+ 66us) service_pool.cc:225] Handling call
1024 19:47:59.546842 (+1835131us) ts_tablet_manager.cc:1151] Deleting tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c
1024 19:47:59.546852 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:59.667820 (+120968us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.681879 (+ 14059us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.683424 (+ 1545us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1834833}
I20251024 19:47:59.683917 1774 tablet_replica.cc:333] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:59.684123 1774 raft_consensus.cc:2243] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:47:59.684602 1774 pending_rounds.cc:70] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:47:59.684792 1774 pending_rounds.cc:77] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429298239852544 op_type: NO_OP noop_request { }
I20251024 19:47:59.684930 1774 raft_consensus.cc:2889] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:47:59.690518 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c (table default.Table_656be68a26b545d2b254abc2edb15254 [id=a1c4569aa6934d4a99787cd9e3240edb]) successfully deleted
I20251024 19:47:59.690110 1882 raft_consensus.cc:493] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:59.692792 1882 raft_consensus.cc:515] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.693504 1882 leader_election.cc:290] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:47:59.685050 1774 raft_consensus.cc:2272] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.694216 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc0bf7aee1a1421b9d985020a1674639" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:47:59.694789 1308 leader_election.cc:343] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:59.695626 1774 ts_tablet_manager.cc:1916] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.710626 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc0bf7aee1a1421b9d985020a1674639" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:59.711944 1766 raft_consensus.cc:359] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:59.713762 1767 tablet_bootstrap.cc:654] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.715165 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 187bc968c2b049028dd25dfb0264c0aa (DEFAULT_TABLE table=default.Table_31fb0d4536b94892aeceb91307c74837 [id=2bd12943c3e24d70a4f02626d7d58f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.716152 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 187bc968c2b049028dd25dfb0264c0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.717730 1621 tablet_service.cc:1505] Processing CreateTablet for tablet e29abfa3bc8944aea7d1acd03e3eb897 (DEFAULT_TABLE table=default.Table_d1b030ba4f784a2b80386b386edec4d9 [id=ba30503223dd4979824ec9c1a50a8ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.718252 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e29abfa3bc8944aea7d1acd03e3eb897. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.719169 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 8ba85164015448d4a688a473cce90c5c (table default.Table_9764ce0360fd4fcfb03ab7df33ed112a [id=0fa321e2d4d847e69435f6de429d8779]). Expected version 1 got 0
I20251024 19:47:59.719889 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet b88a6a5237d146f5b6f082ca3af25b2b from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:59.721874 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f (DEFAULT_TABLE table=default.Table_a1d09325af844d71b290d95b8a4aaf12 [id=b62ec7bcad7c41b1b946e9ebf2f4bf39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.730314 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.731756 1611 tablet_service.cc:1505] Processing CreateTablet for tablet c512ec26e3874cd7869216b1fb90fa54 (DEFAULT_TABLE table=default.Table_a86884fdc4364764911f8b802fc35102 [id=a37609f4cf31400db7b95becebfb4222]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.733676 1766 raft_consensus.cc:385] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251024 19:47:59.733716 1305 leader_election.cc:343] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:47:59.733932 1766 raft_consensus.cc:740] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.733969 1305 leader_election.cc:304] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:47:59.734330 1766 consensus_queue.cc:260] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:59.735371 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f (DEFAULT_TABLE table=default.Table_a1d09325af844d71b290d95b8a4aaf12 [id=b62ec7bcad7c41b1b946e9ebf2f4bf39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.736246 1766 ts_tablet_manager.cc:1434] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.196s user 0.001s sys 0.002s
I20251024 19:47:59.737445 1626 tablet_service.cc:1505] Processing CreateTablet for tablet b1fa439afe5a48a79557019687319af6 (DEFAULT_TABLE table=default.Table_231e4d38da7a4fe4b4e6b33e854aa1d5 [id=77778be7b00c4eccbce4716fc4a864f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.737926 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.738305 1630 tablet_service.cc:1505] Processing CreateTablet for tablet f51c56337ebb450aaf62dcc81ef63ae8 (DEFAULT_TABLE table=default.Table_efcd2af1913c4a73ba5148272aa880aa [id=01a8150e819b4075810a95720eaa5840]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.738431 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 3887fc21726846279fecb02672b9bc07 (DEFAULT_TABLE table=default.Table_e378095ddecc4dcf801a1568160c34b4 [id=450b69bd10cf4c629cc3f8dadc790312]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.738651 1766 tablet_bootstrap.cc:492] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:47:59.738720 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c512ec26e3874cd7869216b1fb90fa54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.739827 1622 tablet_service.cc:1505] Processing CreateTablet for tablet d38154d3ffd34f4b8d7305b1c23a4fae (DEFAULT_TABLE table=default.Table_b15672d0c13d4c939f61e30a2feb96e6 [id=ed879934f5754b92b8cff234f3f66333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.740185 1340 tablet_service.cc:1505] Processing CreateTablet for tablet b1fa439afe5a48a79557019687319af6 (DEFAULT_TABLE table=default.Table_231e4d38da7a4fe4b4e6b33e854aa1d5 [id=77778be7b00c4eccbce4716fc4a864f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.740288 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 3887fc21726846279fecb02672b9bc07 (DEFAULT_TABLE table=default.Table_e378095ddecc4dcf801a1568160c34b4 [id=450b69bd10cf4c629cc3f8dadc790312]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.740667 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1fa439afe5a48a79557019687319af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.741163 1342 tablet_service.cc:1505] Processing CreateTablet for tablet e29abfa3bc8944aea7d1acd03e3eb897 (DEFAULT_TABLE table=default.Table_d1b030ba4f784a2b80386b386edec4d9 [id=ba30503223dd4979824ec9c1a50a8ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.741333 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 187bc968c2b049028dd25dfb0264c0aa (DEFAULT_TABLE table=default.Table_31fb0d4536b94892aeceb91307c74837 [id=2bd12943c3e24d70a4f02626d7d58f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.741521 1348 tablet_service.cc:1505] Processing CreateTablet for tablet f51c56337ebb450aaf62dcc81ef63ae8 (DEFAULT_TABLE table=default.Table_efcd2af1913c4a73ba5148272aa880aa [id=01a8150e819b4075810a95720eaa5840]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.741551 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 505e4fb0755a4caaa462e5f5154bfee2 (DEFAULT_TABLE table=default.Table_873d6a3a39344554b0ca73c575129c36 [id=508d6cdc5d4a494bad31eaf36383e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.741205 1343 tablet_service.cc:1505] Processing CreateTablet for tablet c512ec26e3874cd7869216b1fb90fa54 (DEFAULT_TABLE table=default.Table_a86884fdc4364764911f8b802fc35102 [id=a37609f4cf31400db7b95becebfb4222]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.742012 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 505e4fb0755a4caaa462e5f5154bfee2 (DEFAULT_TABLE table=default.Table_873d6a3a39344554b0ca73c575129c36 [id=508d6cdc5d4a494bad31eaf36383e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.735621 1882 raft_consensus.cc:2749] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:47:59.742269 1345 tablet_service.cc:1505] Processing CreateTablet for tablet d38154d3ffd34f4b8d7305b1c23a4fae (DEFAULT_TABLE table=default.Table_b15672d0c13d4c939f61e30a2feb96e6 [id=ed879934f5754b92b8cff234f3f66333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.742727 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3887fc21726846279fecb02672b9bc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.743127 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 187bc968c2b049028dd25dfb0264c0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.743839 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38154d3ffd34f4b8d7305b1c23a4fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.745704 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e29abfa3bc8944aea7d1acd03e3eb897. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.746042 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c512ec26e3874cd7869216b1fb90fa54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.746284 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51c56337ebb450aaf62dcc81ef63ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.746492 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38154d3ffd34f4b8d7305b1c23a4fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.749236 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51c56337ebb450aaf62dcc81ef63ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.749759 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505e4fb0755a4caaa462e5f5154bfee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.755136 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505e4fb0755a4caaa462e5f5154bfee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.763486 1767 tablet_bootstrap.cc:492] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:59.724881 1775 ts_tablet_manager.cc:1929] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.730077 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 505e4fb0755a4caaa462e5f5154bfee2 (DEFAULT_TABLE table=default.Table_873d6a3a39344554b0ca73c575129c36 [id=508d6cdc5d4a494bad31eaf36383e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.766462 1775 log.cc:1199] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/e2d97caee586418a8981108465f6b704
I20251024 19:47:59.766803 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505e4fb0755a4caaa462e5f5154bfee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.766885 1775 ts_tablet_manager.cc:1950] T e2d97caee586418a8981108465f6b704 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:59.769196 1766 tablet_bootstrap.cc:654] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.730423 1489 tablet_service.cc:1505] Processing CreateTablet for tablet f51c56337ebb450aaf62dcc81ef63ae8 (DEFAULT_TABLE table=default.Table_efcd2af1913c4a73ba5148272aa880aa [id=01a8150e819b4075810a95720eaa5840]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.769944 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51c56337ebb450aaf62dcc81ef63ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.730505 1480 tablet_service.cc:1505] Processing CreateTablet for tablet d38154d3ffd34f4b8d7305b1c23a4fae (DEFAULT_TABLE table=default.Table_b15672d0c13d4c939f61e30a2feb96e6 [id=ed879934f5754b92b8cff234f3f66333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.731243 1484 tablet_service.cc:1505] Processing CreateTablet for tablet c512ec26e3874cd7869216b1fb90fa54 (DEFAULT_TABLE table=default.Table_a86884fdc4364764911f8b802fc35102 [id=a37609f4cf31400db7b95becebfb4222]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.731762 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f (DEFAULT_TABLE table=default.Table_a1d09325af844d71b290d95b8a4aaf12 [id=b62ec7bcad7c41b1b946e9ebf2f4bf39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.773931 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38154d3ffd34f4b8d7305b1c23a4fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.732137 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 187bc968c2b049028dd25dfb0264c0aa (DEFAULT_TABLE table=default.Table_31fb0d4536b94892aeceb91307c74837 [id=2bd12943c3e24d70a4f02626d7d58f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.774278 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 187bc968c2b049028dd25dfb0264c0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.774930 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c512ec26e3874cd7869216b1fb90fa54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.781813 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.732527 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 3887fc21726846279fecb02672b9bc07 (DEFAULT_TABLE table=default.Table_e378095ddecc4dcf801a1568160c34b4 [id=450b69bd10cf4c629cc3f8dadc790312]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.782558 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet e2d97caee586418a8981108465f6b704 (table default.Table_95644b1e2fa0449d806efb9c458266b9 [id=168a2adf03184a04b7e8f5017919a08c]) successfully deleted
I20251024 19:47:59.732908 1490 tablet_service.cc:1505] Processing CreateTablet for tablet b1fa439afe5a48a79557019687319af6 (DEFAULT_TABLE table=default.Table_231e4d38da7a4fe4b4e6b33e854aa1d5 [id=77778be7b00c4eccbce4716fc4a864f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.733331 1488 tablet_service.cc:1505] Processing CreateTablet for tablet e29abfa3bc8944aea7d1acd03e3eb897 (DEFAULT_TABLE table=default.Table_d1b030ba4f784a2b80386b386edec4d9 [id=ba30503223dd4979824ec9c1a50a8ae2]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:47:59.786159 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 5fd7726c532145c6b44ea9a37544510d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fd7726c532145c6b44ea9a37544510d already in progress: creating tablet
W20251024 19:47:59.786754 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet ca0527af718f494a8bbcc66e7b325bed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca0527af718f494a8bbcc66e7b325bed already in progress: creating tablet
I20251024 19:47:59.786811 1774 ts_tablet_manager.cc:1929] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:47:59.786942 1774 log.cc:1199] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/922f975a672746b3b72acf4d3050f688
W20251024 19:47:59.787281 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 73ffe342f541447d8519ee24bc98a0f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73ffe342f541447d8519ee24bc98a0f3 already in progress: creating tablet
I20251024 19:47:59.788118 1774 ts_tablet_manager.cc:1950] T 922f975a672746b3b72acf4d3050f688 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:59.788517 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 119) took 2076 ms. Trace:
I20251024 19:47:59.788777 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3887fc21726846279fecb02672b9bc07. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:47:59.789537 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7c357a2a81064b3891afda6d252898b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c357a2a81064b3891afda6d252898b9 already in progress: creating tablet
W20251024 19:47:59.793413 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b6661b9e9a174849b6093d09cd8c1251 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6661b9e9a174849b6093d09cd8c1251 already in progress: creating tablet
I20251024 19:47:59.795212 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1fa439afe5a48a79557019687319af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.797201 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e29abfa3bc8944aea7d1acd03e3eb897. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.800925 1774 rpcz_store.cc:276] 1024 19:47:57.712260 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.718319 (+ 6059us) service_pool.cc:225] Handling call
1024 19:47:59.683855 (+1965536us) ts_tablet_manager.cc:1151] Deleting tablet 922f975a672746b3b72acf4d3050f688
1024 19:47:59.683866 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:59.703957 (+ 20091us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.786696 (+ 82739us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.788487 (+ 1791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1959668}
I20251024 19:47:59.803337 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 112) took 2090 ms. Trace:
I20251024 19:47:59.805152 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1fa439afe5a48a79557019687319af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.805338 1775 tablet_replica.cc:333] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:59.805557 1775 raft_consensus.cc:2243] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.805784 1775 raft_consensus.cc:2272] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.806507 1775 ts_tablet_manager.cc:1916] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.814199 1774 rpcz_store.cc:276] 1024 19:47:57.711425 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.718824 (+ 7399us) service_pool.cc:225] Handling call
1024 19:47:59.801562 (+2082738us) ts_tablet_manager.cc:1151] Deleting tablet f05ad092936b4bafa3ae795904c2df78
1024 19:47:59.801598 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:59.802135 (+ 537us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2072567}
I20251024 19:47:59.814977 1774 tablet_replica.cc:333] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:59.817641 1774 raft_consensus.cc:2243] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.817956 1774 raft_consensus.cc:2272] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:47:59.818775 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9fffc42ddda14bd28800917ff2f0ced5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fffc42ddda14bd28800917ff2f0ced5 already in progress: creating tablet
I20251024 19:47:59.834035 1767 ts_tablet_manager.cc:1403] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.100s user 0.014s sys 0.000s
I20251024 19:47:59.835954 1774 ts_tablet_manager.cc:1916] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.841754 1767 raft_consensus.cc:359] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:59.842658 1767 raft_consensus.cc:385] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.842840 1767 raft_consensus.cc:740] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.843240 1767 consensus_queue.cc:260] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:59.842478 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f1ad65fd080e452e8094975d420336c7
I20251024 19:47:59.847654 1183 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]). Expected version 3 got 0
I20251024 19:47:59.848729 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_06b07944861041f8a66001a6555d7659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:59.843683 1190 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 8ba85164015448d4a688a473cce90c5c (table default.Table_9764ce0360fd4fcfb03ab7df33ed112a [id=0fa321e2d4d847e69435f6de429d8779]). Expected version 1 got 0
I20251024 19:47:59.851018 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_07aae1757a3444a4bbb8d729d6f08a1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.851656 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07aae1757a3444a4bbb8d729d6f08a1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:59.849517 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06b07944861041f8a66001a6555d7659 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:59.855386 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9623d7baf0174a019e2353a5effef081 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9623d7baf0174a019e2353a5effef081 already in progress: creating tablet
I20251024 19:47:59.856065 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 922f975a672746b3b72acf4d3050f688 (table default.Table_ab42d318a6cc4626be3d62dda5948bf0 [id=eab377d5ce1343d7a12c3740fe46beb5]) successfully deleted
W20251024 19:47:59.856617 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet f05ad092936b4bafa3ae795904c2df78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f05ad092936b4bafa3ae795904c2df78 already in progress: creating tablet
I20251024 19:47:59.857594 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8456d5c39d4b4ac985982e8eadfa2750" } new_table_name: "default.Table_a1366192b3134e3abfc231729cb42344" modify_external_catalogs: true
I20251024 19:47:59.858578 1768 tablet_bootstrap.cc:492] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:59.858892 1768 ts_tablet_manager.cc:1403] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.216s user 0.016s sys 0.016s
I20251024 19:47:59.859983 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2037fcca87b94b5fb31639b4d07307b0
I20251024 19:47:59.860409 1186 catalog_manager.cc:6719] ReplaceTablet: tablet 2037fcca87b94b5fb31639b4d07307b0 deleted and replaced by tablet 5d129bdc876e425aabcf1c99997e319a
I20251024 19:47:59.860695 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_bc5c06e7b159482c89d22461fe071b3f" } new_table_name: "default.Table_78e6806b2ede4763843892de1ab015e3" modify_external_catalogs: true
I20251024 19:47:59.860615 1768 raft_consensus.cc:359] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.860954 1768 raft_consensus.cc:385] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.861052 1768 raft_consensus.cc:740] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.861982 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet aebf76d4a66947ed9756643740418798
I20251024 19:47:59.862411 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet f1ad65fd080e452e8094975d420336c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.861464 1768 consensus_queue.cc:260] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.862602 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_46251d3ce016440287b2a378eae2e9a7" } new_table_name: "default.Table_f37eb464f5144d1a8135a5f1a44cee4f" modify_external_catalogs: true
I20251024 19:47:59.863358 1768 ts_tablet_manager.cc:1434] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:47:59.863804 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 27b32126d04e480f8db7b2b20aba5873
I20251024 19:47:59.864424 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5e63f20b2f66459795a5289f47ffaf0e" } new_table_name: "default.Table_4a085504a0884b37b6787b3bcf7a847f" modify_external_catalogs: true
I20251024 19:47:59.865314 1768 tablet_bootstrap.cc:492] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:59.865778 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 50d919a0b3d74e9a97d136a84d50e24b
I20251024 19:47:59.867751 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_96f7d0ed2ff04485945a33b8eda07570" } new_table_name: "default.Table_71bcb15d38b842efbdfc0fb90171e71e" modify_external_catalogs: true
I20251024 19:47:59.868867 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b88a6a5237d146f5b6f082ca3af25b2b
I20251024 19:47:59.869306 1191 catalog_manager.cc:6719] ReplaceTablet: tablet b88a6a5237d146f5b6f082ca3af25b2b deleted and replaced by tablet 639836c656f44a819036f4d00e055bd9
I20251024 19:47:59.869642 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5006c140e0964b569953c4f03495c697"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.870177 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5006c140e0964b569953c4f03495c697 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.870203 1767 ts_tablet_manager.cc:1434] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20251024 19:47:59.872406 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet fca73ab61f0c47df9186459caf2b9a32
I20251024 19:47:59.872874 1192 catalog_manager.cc:6719] ReplaceTablet: tablet fca73ab61f0c47df9186459caf2b9a32 deleted and replaced by tablet f83b9290f00145a780318a43249eec12
I20251024 19:47:59.873163 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_83162e45d21e4d47a1cb086b20db8ef3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:59.873642 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 44ee03ede02c4bdab706fde910978332
I20251024 19:47:59.874320 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_646390b8820e4af996e1fa10cf39ac39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:59.874415 1767 tablet_bootstrap.cc:492] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:47:59.874780 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_646390b8820e4af996e1fa10cf39ac39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.875969 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet b88a6a5237d146f5b6f082ca3af25b2b with delete_type TABLET_DATA_DELETED (replaced by tablet 639836c656f44a819036f4d00e055bd9) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.876158 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 50d919a0b3d74e9a97d136a84d50e24b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.879706 1819 raft_consensus.cc:493] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:47:59.879902 1819 raft_consensus.cc:515] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:47:59.881322 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83162e45d21e4d47a1cb086b20db8ef3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.881330 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet aebf76d4a66947ed9756643740418798 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.872170 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet aebf76d4a66947ed9756643740418798 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.877707 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet 27b32126d04e480f8db7b2b20aba5873 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.879544 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 50d919a0b3d74e9a97d136a84d50e24b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.880336 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet b88a6a5237d146f5b6f082ca3af25b2b with delete_type TABLET_DATA_DELETED (replaced by tablet 639836c656f44a819036f4d00e055bd9) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.881174 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet fca73ab61f0c47df9186459caf2b9a32 with delete_type TABLET_DATA_DELETED (replaced by tablet f83b9290f00145a780318a43249eec12) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.884701 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 44ee03ede02c4bdab706fde910978332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.869901 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 2037fcca87b94b5fb31639b4d07307b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d129bdc876e425aabcf1c99997e319a) from {username='slave'} at 127.0.0.1:54174
I20251024 19:47:59.867734 1768 tablet_bootstrap.cc:654] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.888020 1774 ts_tablet_manager.cc:1929] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.888268 1774 log.cc:1199] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/110907ae943f45b5bcaa9b66ccdb84fd
I20251024 19:47:59.889142 1774 ts_tablet_manager.cc:1950] T 110907ae943f45b5bcaa9b66ccdb84fd P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:59.890383 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 111) took 2178 ms. Trace:
I20251024 19:47:59.890615 1774 rpcz_store.cc:276] 1024 19:47:57.711268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.718898 (+ 7630us) service_pool.cc:225] Handling call
1024 19:47:59.814839 (+2095941us) ts_tablet_manager.cc:1151] Deleting tablet 110907ae943f45b5bcaa9b66ccdb84fd
1024 19:47:59.814850 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:59.840320 (+ 25470us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.887873 (+ 47553us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.890107 (+ 2234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2078529}
I20251024 19:47:59.891296 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 110907ae943f45b5bcaa9b66ccdb84fd (table default.Table_2cbb0aad326e4bae85d4c5cb833150c6 [id=1acdc5d0bfb94e0faf47055b20d803b0]) successfully deleted
I20251024 19:47:59.891803 1774 tablet_replica.cc:333] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:59.892130 1774 raft_consensus.cc:2243] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.892652 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet f1ad65fd080e452e8094975d420336c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.893335 1774 raft_consensus.cc:2272] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.894204 1774 ts_tablet_manager.cc:1916] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.894706 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 27b32126d04e480f8db7b2b20aba5873 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.897369 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 2037fcca87b94b5fb31639b4d07307b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d129bdc876e425aabcf1c99997e319a) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.898526 1776 ts_tablet_manager.cc:1929] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.899295 1776 log.cc:1199] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/230606a0ce084b7bab0e41e48cf8fa89
I20251024 19:47:59.905995 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 50d919a0b3d74e9a97d136a84d50e24b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.907622 1481 tablet_service.cc:1552] Processing DeleteTablet for tablet b88a6a5237d146f5b6f082ca3af25b2b with delete_type TABLET_DATA_DELETED (replaced by tablet 639836c656f44a819036f4d00e055bd9) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.908463 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb08e56db59b4db796c50db5bda93af9" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:47:59.908893 1371 raft_consensus.cc:2468] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:47:59.913852 1774 ts_tablet_manager.cc:1929] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.914321 1776 ts_tablet_manager.cc:1950] T 230606a0ce084b7bab0e41e48cf8fa89 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:47:59.914575 1774 log.cc:1199] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/cc0bf7aee1a1421b9d985020a1674639
I20251024 19:47:59.914758 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet fca73ab61f0c47df9186459caf2b9a32 with delete_type TABLET_DATA_DELETED (replaced by tablet f83b9290f00145a780318a43249eec12) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.915622 1774 ts_tablet_manager.cc:1950] T cc0bf7aee1a1421b9d985020a1674639 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:47:59.914815 1767 tablet_bootstrap.cc:654] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.916335 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 44ee03ede02c4bdab706fde910978332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:47:59.916759 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 115) took 2205 ms. Trace:
I20251024 19:47:59.916747 1480 tablet_service.cc:1552] Processing DeleteTablet for tablet aebf76d4a66947ed9756643740418798 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.916921 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 2037fcca87b94b5fb31639b4d07307b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d129bdc876e425aabcf1c99997e319a) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.916880 1774 rpcz_store.cc:276] 1024 19:47:57.711745 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.718874 (+ 7129us) service_pool.cc:225] Handling call
1024 19:47:59.891038 (+2172164us) ts_tablet_manager.cc:1151] Deleting tablet cc0bf7aee1a1421b9d985020a1674639
1024 19:47:59.891048 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:59.901793 (+ 10745us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.913713 (+ 11920us) tablet_metadata.cc:641] Metadata flushed
1024 19:47:59.916722 (+ 3009us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2150046}
I20251024 19:47:59.917034 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 27b32126d04e480f8db7b2b20aba5873 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.917349 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 127) took 2146 ms. Trace:
I20251024 19:47:59.917466 1774 rpcz_store.cc:276] 1024 19:47:57.770536 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.776728 (+ 6192us) service_pool.cc:225] Handling call
1024 19:47:59.917183 (+2140455us) ts_tablet_manager.cc:1151] Deleting tablet b9b1e0338fd14634a1e921735ae61289
1024 19:47:59.917195 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:59.917308 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2140163}
I20251024 19:47:59.917742 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet cc0bf7aee1a1421b9d985020a1674639 (table default.Table_272323b2c4ac4c8cb0ff412a632ee43d [id=48bed8c74a444f9fa83d52387a2650c5]) successfully deleted
I20251024 19:47:59.918277 1819 leader_election.cc:290] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:59.919736 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 113) took 2208 ms. Trace:
I20251024 19:47:59.920261 1774 rpcz_store.cc:276] 1024 19:47:57.711555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.718921 (+ 7366us) service_pool.cc:225] Handling call
1024 19:47:59.919599 (+2200678us) ts_tablet_manager.cc:1151] Deleting tablet 31647a48fb834c80b6920c756a94250d
1024 19:47:59.919608 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:59.919704 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2122139}
I20251024 19:47:59.920815 1774 tablet_replica.cc:333] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:47:59.921305 1774 raft_consensus.cc:2243] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:47:59.921818 1774 raft_consensus.cc:2272] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.922094 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet fca73ab61f0c47df9186459caf2b9a32 with delete_type TABLET_DATA_DELETED (replaced by tablet f83b9290f00145a780318a43249eec12) from {username='slave'} at 127.0.0.1:46462
W20251024 19:47:59.922839 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b9b1e0338fd14634a1e921735ae61289 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9b1e0338fd14634a1e921735ae61289 already in progress: creating tablet
I20251024 19:47:59.923027 1774 ts_tablet_manager.cc:1916] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:47:59.923166 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 31647a48fb834c80b6920c756a94250d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31647a48fb834c80b6920c756a94250d already in progress: creating tablet
I20251024 19:47:59.923710 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 44ee03ede02c4bdab706fde910978332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.923985 1766 tablet_bootstrap.cc:492] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:47:59.925024 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb08e56db59b4db796c50db5bda93af9" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:47:59.925307 1649 raft_consensus.cc:2468] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:47:59.930745 1766 ts_tablet_manager.cc:1403] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.188s user 0.044s sys 0.040s
I20251024 19:47:59.931340 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet f1ad65fd080e452e8094975d420336c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.926266 1447 leader_election.cc:304] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:47:59.934633 1819 raft_consensus.cc:2804] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:47:59.934756 1819 raft_consensus.cc:493] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:47:59.935063 1819 raft_consensus.cc:3060] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:59.935729 1766 raft_consensus.cc:359] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:59.938366 1819 raft_consensus.cc:515] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:59.942322 1768 tablet_bootstrap.cc:492] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:59.942498 1768 ts_tablet_manager.cc:1403] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.006s
I20251024 19:47:59.943359 1768 raft_consensus.cc:359] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.943537 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb08e56db59b4db796c50db5bda93af9" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:47:59.943621 1768 raft_consensus.cc:385] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.943723 1371 raft_consensus.cc:3060] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:59.945906 1768 raft_consensus.cc:740] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.946625 1768 consensus_queue.cc:260] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.947513 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f57b8c8910c2489a80979e2ffe453f3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:47:59.948446 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 230606a0ce084b7bab0e41e48cf8fa89 (table default.Table_ebef5e3474bb4a288fcbaf6def6a7ce1 [id=469c675d15e3482d98671ee6b9e4bd42]) successfully deleted
W20251024 19:47:59.948776 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b88a6a5237d146f5b6f082ca3af25b2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b88a6a5237d146f5b6f082ca3af25b2b already in progress: creating tablet
I20251024 19:47:59.949052 1768 ts_tablet_manager.cc:1434] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251024 19:47:59.949508 1371 raft_consensus.cc:2468] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
W20251024 19:47:59.950632 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f57b8c8910c2489a80979e2ffe453f3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.950505 1775 ts_tablet_manager.cc:1929] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:47:59.951783 1775 log.cc:1199] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/97c6dce647064344b0b53a7ffaeafb75
I20251024 19:47:59.952430 1775 ts_tablet_manager.cc:1950] T 97c6dce647064344b0b53a7ffaeafb75 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:47:59.956123 1766 raft_consensus.cc:385] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:47:59.956254 1766 raft_consensus.cc:740] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:47:59.956549 1766 consensus_queue.cc:260] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:59.956910 1819 leader_election.cc:290] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:47:59.960119 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb08e56db59b4db796c50db5bda93af9" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:47:59.960393 1649 raft_consensus.cc:3060] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:47:59.950845 1768 tablet_bootstrap.cc:492] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:47:59.964470 1447 leader_election.cc:304] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:47:59.964740 1819 raft_consensus.cc:2804] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:47:59.964884 1819 raft_consensus.cc:697] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:47:59.965173 1819 consensus_queue.cc:237] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:47:59.966466 1768 tablet_bootstrap.cc:654] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:47:59.953650 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 97c6dce647064344b0b53a7ffaeafb75 (table default.Table_85e6c4c107c94e7abfc7c3396aaf6e81 [id=1b078c2814dd4ff99c1e2ee01d0159cf]) successfully deleted
W20251024 19:47:59.957371 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a76d61c3464b41519d7657796bd853ff consensus state has no leader
W20251024 19:47:59.967805 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 50d919a0b3d74e9a97d136a84d50e24b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50d919a0b3d74e9a97d136a84d50e24b already in progress: creating tablet
W20251024 19:47:59.968081 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b88a6a5237d146f5b6f082ca3af25b2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b88a6a5237d146f5b6f082ca3af25b2b already in progress: creating tablet
W20251024 19:47:59.968281 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet aebf76d4a66947ed9756643740418798 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aebf76d4a66947ed9756643740418798 already in progress: creating tablet
W20251024 19:47:59.968506 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2037fcca87b94b5fb31639b4d07307b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2037fcca87b94b5fb31639b4d07307b0 already in progress: creating tablet
W20251024 19:47:59.968717 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 27b32126d04e480f8db7b2b20aba5873 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27b32126d04e480f8db7b2b20aba5873 already in progress: creating tablet
W20251024 19:47:59.969095 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet fca73ab61f0c47df9186459caf2b9a32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fca73ab61f0c47df9186459caf2b9a32 already in progress: creating tablet
W20251024 19:47:59.958487 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe97e31310ae4733ba93de1b62a37842 consensus state has no leader
I20251024 19:47:59.970088 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5a60ae2f44c2462086e73f80ad013457"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.970606 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a60ae2f44c2462086e73f80ad013457 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:47:59.971905 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b89a90bb4ca447fbaf34fe39001c261 consensus state has no leader
I20251024 19:47:59.972203 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet 0a12e415590a4c71aab4ad7d30412c83 from {username='slave'} at 127.0.0.1:51416
I20251024 19:47:59.961725 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet a76d61c3464b41519d7657796bd853ff from {username='slave'} at 127.0.0.1:51416
W20251024 19:47:59.972643 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9bf08ccf7054fedb9efcbb00354c71a consensus state has no leader
I20251024 19:47:59.972994 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2f543df6b1194a7195ecd656134e4b17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:47:59.973887 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7352d84226db467dad31ae5941842b80 consensus state has no leader
I20251024 19:47:59.974759 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3887fc21726846279fecb02672b9bc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.975252 1776 tablet_replica.cc:333] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:47:59.978684 1768 tablet_bootstrap.cc:492] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:47:59.978829 1775 tablet_replica.cc:333] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:47:59.979049 1775 raft_consensus.cc:2243] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
W20251024 19:47:59.973531 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f543df6b1194a7195ecd656134e4b17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:47:59.979364 1775 raft_consensus.cc:2272] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:47:59.964246 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 50d919a0b3d74e9a97d136a84d50e24b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50d919a0b3d74e9a97d136a84d50e24b already in progress: creating tablet
W20251024 19:47:59.984400 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet aebf76d4a66947ed9756643740418798 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aebf76d4a66947ed9756643740418798 already in progress: creating tablet
I20251024 19:47:59.987181 1768 ts_tablet_manager.cc:1403] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.004s
I20251024 19:47:59.987228 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 608696ec90ed45e1ad939ed2bfc88d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:47:59.988747 1346 tablet_service.cc:1505] Processing CreateTablet for tablet be8126459ef541f39bc133f27c7d0f59 (DEFAULT_TABLE table=default.Table_0d796a82cf4b4e0f9cba35051c69293d [id=06db25030489409aa797ab8ea05dd6c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.989485 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be8126459ef541f39bc133f27c7d0f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.990409 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 16bc737104ab46b3b071a0760dbc847d (DEFAULT_TABLE table=default.Table_775e17f58fef4f2b850035a03e411a08 [id=6239ceae506442b79bbe79c48813b49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.990447 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 16bc737104ab46b3b071a0760dbc847d (DEFAULT_TABLE table=default.Table_775e17f58fef4f2b850035a03e411a08 [id=6239ceae506442b79bbe79c48813b49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.990847 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16bc737104ab46b3b071a0760dbc847d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.990932 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16bc737104ab46b3b071a0760dbc847d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.991765 1768 raft_consensus.cc:359] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:47:59.992519 1775 ts_tablet_manager.cc:1916] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:47:59.997023 1767 tablet_bootstrap.cc:492] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:47:59.997136 1490 tablet_service.cc:1505] Processing CreateTablet for tablet be8126459ef541f39bc133f27c7d0f59 (DEFAULT_TABLE table=default.Table_0d796a82cf4b4e0f9cba35051c69293d [id=06db25030489409aa797ab8ea05dd6c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:47:59.997617 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be8126459ef541f39bc133f27c7d0f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:47:59.997632 1776 raft_consensus.cc:2243] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:47:59.998004 1649 raft_consensus.cc:2468] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:47:59.999485 1776 raft_consensus.cc:2272] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:47:59.999853 1767 ts_tablet_manager.cc:1403] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.128s user 0.012s sys 0.011s
I20251024 19:48:00.000540 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet fb17a35d8d8446cf9870d05e107ed237 with delete_type TABLET_DATA_DELETED (replaced by tablet ca0527af718f494a8bbcc66e7b325bed) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.001075 1611 tablet_service.cc:1505] Processing CreateTablet for tablet be8126459ef541f39bc133f27c7d0f59 (DEFAULT_TABLE table=default.Table_0d796a82cf4b4e0f9cba35051c69293d [id=06db25030489409aa797ab8ea05dd6c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.001637 1776 ts_tablet_manager.cc:1916] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.004016 1767 raft_consensus.cc:359] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.004266 1767 raft_consensus.cc:385] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.004383 1767 raft_consensus.cc:740] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.004581 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 0a12e415590a4c71aab4ad7d30412c83
I20251024 19:48:00.005478 1184 catalog_manager.cc:6719] ReplaceTablet: tablet 0a12e415590a4c71aab4ad7d30412c83 deleted and replaced by tablet d6de2c21796f4bc4904f88b654c3baec
I20251024 19:48:00.006247 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet a76d61c3464b41519d7657796bd853ff
I20251024 19:48:00.007207 1183 catalog_manager.cc:6719] ReplaceTablet: tablet a76d61c3464b41519d7657796bd853ff deleted and replaced by tablet b240965605904533bd2af99f61aaef26
I20251024 19:48:00.008071 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet a76d61c3464b41519d7657796bd853ff with delete_type TABLET_DATA_DELETED (replaced by tablet b240965605904533bd2af99f61aaef26) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:00.009879 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 44ee03ede02c4bdab706fde910978332 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ee03ede02c4bdab706fde910978332 already in progress: creating tablet
I20251024 19:48:00.009955 1767 consensus_queue.cc:260] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.011898 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a12e415590a4c71aab4ad7d30412c83 with delete_type TABLET_DATA_DELETED (replaced by tablet d6de2c21796f4bc4904f88b654c3baec) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.011960 1767 ts_tablet_manager.cc:1434] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20251024 19:48:00.012290 1767 tablet_bootstrap.cc:492] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:00.012454 1768 raft_consensus.cc:385] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.012585 1768 raft_consensus.cc:740] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.012821 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet a76d61c3464b41519d7657796bd853ff with delete_type TABLET_DATA_DELETED (replaced by tablet b240965605904533bd2af99f61aaef26) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.012874 1768 consensus_queue.cc:260] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.013222 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a12e415590a4c71aab4ad7d30412c83 with delete_type TABLET_DATA_DELETED (replaced by tablet d6de2c21796f4bc4904f88b654c3baec) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:00.013660 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet a76d61c3464b41519d7657796bd853ff with delete_type TABLET_DATA_DELETED (replaced by tablet b240965605904533bd2af99f61aaef26) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:00.015033 1774 ts_tablet_manager.cc:1929] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:00.016148 1768 ts_tablet_manager.cc:1434] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20251024 19:48:00.020104 1774 log.cc:1199] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/608696ec90ed45e1ad939ed2bfc88d34
I20251024 19:48:00.020879 1774 ts_tablet_manager.cc:1950] T 608696ec90ed45e1ad939ed2bfc88d34 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:00.022573 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 130) took 2227 ms. Trace:
I20251024 19:48:00.020413 1768 tablet_bootstrap.cc:492] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:00.026497 1768 tablet_bootstrap.cc:654] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.028007 1774 rpcz_store.cc:276] 1024 19:47:57.794904 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.794978 (+ 74us) service_pool.cc:225] Handling call
1024 19:47:59.920752 (+2125774us) ts_tablet_manager.cc:1151] Deleting tablet 608696ec90ed45e1ad939ed2bfc88d34
1024 19:47:59.920761 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:47:59.982706 (+ 61945us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.014903 (+ 32197us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.022508 (+ 7605us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2123102}
I20251024 19:48:00.029410 1774 tablet_replica.cc:333] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:00.029606 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 608696ec90ed45e1ad939ed2bfc88d34 (table default.Table_6e1fbc007b464e8c93803ba262a19fea [id=2c92cb09b287410c90b811e4b88c2260]) successfully deleted
I20251024 19:48:00.030678 1774 raft_consensus.cc:2243] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:00.031697 1774 raft_consensus.cc:2272] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:00.032583 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a12e415590a4c71aab4ad7d30412c83 with delete_type TABLET_DATA_DELETED (replaced by tablet d6de2c21796f4bc4904f88b654c3baec) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.034086 1774 ts_tablet_manager.cc:1916] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.034590 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 16bc737104ab46b3b071a0760dbc847d (DEFAULT_TABLE table=default.Table_775e17f58fef4f2b850035a03e411a08 [id=6239ceae506442b79bbe79c48813b49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.035347 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be8126459ef541f39bc133f27c7d0f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.037468 1767 tablet_bootstrap.cc:654] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.045792 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 3caecdff61ce48f7b7c3846bb380d21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.048411 1775 ts_tablet_manager.cc:1929] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251024 19:48:00.048651 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16bc737104ab46b3b071a0760dbc847d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.059621 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_719145aefd754122ba1bb0476ca4ac7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.060256 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_719145aefd754122ba1bb0476ca4ac7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:00.060701 1305 consensus_peers.cc:597] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65f646bd05274cb6859c0b7f92b61352. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:00.065007 1774 ts_tablet_manager.cc:1929] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:00.065400 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_231e4d38da7a4fe4b4e6b33e854aa1d5" } modify_external_catalogs: true
I20251024 19:48:00.065742 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_231e4d38da7a4fe4b4e6b33e854aa1d5" } modify_external_catalogs: true
I20251024 19:48:00.066259 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a5d11f93359546078ef5becf10539131"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.066782 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5d11f93359546078ef5becf10539131 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.066989 1774 log.cc:1199] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/230606a0ce084b7bab0e41e48cf8fa89
I20251024 19:48:00.067386 1774 ts_tablet_manager.cc:1950] T 230606a0ce084b7bab0e41e48cf8fa89 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:00.070318 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 131) took 2272 ms. Trace:
I20251024 19:48:00.070494 1774 rpcz_store.cc:276] 1024 19:47:57.797349 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.797395 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:00.029309 (+2231914us) ts_tablet_manager.cc:1151] Deleting tablet 230606a0ce084b7bab0e41e48cf8fa89
1024 19:48:00.029319 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:00.039557 (+ 10238us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.064863 (+ 25306us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.070024 (+ 5161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2230021}
I20251024 19:48:00.071146 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 230606a0ce084b7bab0e41e48cf8fa89 (table default.Table_ebef5e3474bb4a288fcbaf6def6a7ce1 [id=469c675d15e3482d98671ee6b9e4bd42]) successfully deleted
I20251024 19:48:00.071653 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 129) took 2277 ms. Trace:
I20251024 19:48:00.071835 1774 rpcz_store.cc:276] 1024 19:47:57.794204 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.795823 (+ 1619us) service_pool.cc:225] Handling call
1024 19:48:00.071473 (+2275650us) ts_tablet_manager.cc:1151] Deleting tablet 8a1f1892bd3b4776adc7c2519fcdf411
1024 19:48:00.071486 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:00.071610 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2272082}
W20251024 19:48:00.072439 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 8a1f1892bd3b4776adc7c2519fcdf411 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a1f1892bd3b4776adc7c2519fcdf411 already in progress: creating tablet
I20251024 19:48:00.072798 1626 tablet_service.cc:1505] Processing CreateTablet for tablet edb49884389c4dba957f0ee5af4ae0e4 (DEFAULT_TABLE table=default.Table_07aae1757a3444a4bbb8d729d6f08a1d [id=b410120363fe4ba5907be2c8793b5e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.073378 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edb49884389c4dba957f0ee5af4ae0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.073827 1775 log.cc:1199] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/f1ad65fd080e452e8094975d420336c7
I20251024 19:48:00.074184 1352 tablet_service.cc:1505] Processing CreateTablet for tablet edb49884389c4dba957f0ee5af4ae0e4 (DEFAULT_TABLE table=default.Table_07aae1757a3444a4bbb8d729d6f08a1d [id=b410120363fe4ba5907be2c8793b5e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.074306 1775 ts_tablet_manager.cc:1950] T f1ad65fd080e452e8094975d420336c7 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:00.074556 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edb49884389c4dba957f0ee5af4ae0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.077188 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 639836c656f44a819036f4d00e055bd9 (DEFAULT_TABLE table=default.Table_6eb10ae51f3a43c2ab15f362708fba94 [id=f3080df28ee94876bb6b790cea45a4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.077555 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639836c656f44a819036f4d00e055bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.071195 1184 catalog_manager.cc:5649] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:00.079131 1184 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]). Expected version 3 got 0
I20251024 19:48:00.080585 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 639836c656f44a819036f4d00e055bd9 (DEFAULT_TABLE table=default.Table_6eb10ae51f3a43c2ab15f362708fba94 [id=f3080df28ee94876bb6b790cea45a4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.080721 1484 tablet_service.cc:1505] Processing CreateTablet for tablet c82f60819e7a442bbc0cc04659e14e87 (DEFAULT_TABLE table=default.Table_6f376570019346199a3c8c77d879fa1c [id=c5f7c2469f394bba8b91e2f20170963e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.080982 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639836c656f44a819036f4d00e055bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.081781 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c82f60819e7a442bbc0cc04659e14e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.083962 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 9489a01434a24d9ca94791252950725c (DEFAULT_TABLE table=default.Table_1c89d244042a4852a9cb4113f6726332 [id=ba207dd9990d4a0e85eaefbb4908dd02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.084471 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9489a01434a24d9ca94791252950725c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.085862 1482 tablet_service.cc:1505] Processing CreateTablet for tablet f83b9290f00145a780318a43249eec12 (DEFAULT_TABLE table=default.Table_dd9b9ef96622439ea016c3813fcbb81a [id=be1cc9d047554ea6a9afa531fb08a81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.086244 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f83b9290f00145a780318a43249eec12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.088840 1616 tablet_service.cc:1505] Processing CreateTablet for tablet f83b9290f00145a780318a43249eec12 (DEFAULT_TABLE table=default.Table_dd9b9ef96622439ea016c3813fcbb81a [id=be1cc9d047554ea6a9afa531fb08a81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.089218 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f83b9290f00145a780318a43249eec12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.089854 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 5d129bdc876e425aabcf1c99997e319a (DEFAULT_TABLE table=default.Table_9edfe580f57e43919a206badbc0bbfd6 [id=d28fd80220c64f8496a09f5d4e712874]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.090369 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d129bdc876e425aabcf1c99997e319a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.091238 1767 tablet_bootstrap.cc:492] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:00.091411 1767 ts_tablet_manager.cc:1403] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.004s
I20251024 19:48:00.092291 1767 raft_consensus.cc:359] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.092721 1767 raft_consensus.cc:385] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.092808 1767 raft_consensus.cc:740] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.093075 1767 consensus_queue.cc:260] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.093665 1767 ts_tablet_manager.cc:1434] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251024 19:48:00.093988 1767 tablet_bootstrap.cc:492] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:00.058300 1490 tablet_service.cc:1505] Processing CreateTablet for tablet edb49884389c4dba957f0ee5af4ae0e4 (DEFAULT_TABLE table=default.Table_07aae1757a3444a4bbb8d729d6f08a1d [id=b410120363fe4ba5907be2c8793b5e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.095158 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edb49884389c4dba957f0ee5af4ae0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.066008 1766 ts_tablet_manager.cc:1434] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.135s user 0.004s sys 0.016s
I20251024 19:48:00.073439 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 9489a01434a24d9ca94791252950725c (DEFAULT_TABLE table=default.Table_1c89d244042a4852a9cb4113f6726332 [id=ba207dd9990d4a0e85eaefbb4908dd02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.098086 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9489a01434a24d9ca94791252950725c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.079523 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet f1ad65fd080e452e8094975d420336c7 (table default.Table_a54ffafa18b848b79a290f58c9754bec [id=2fd5c541b06a46469e65ed0942d2c394]) successfully deleted
W20251024 19:48:00.099826 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 608696ec90ed45e1ad939ed2bfc88d34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 608696ec90ed45e1ad939ed2bfc88d34 already in progress: creating tablet
W20251024 19:48:00.100090 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 0a12e415590a4c71aab4ad7d30412c83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a12e415590a4c71aab4ad7d30412c83 already in progress: creating tablet
W20251024 19:48:00.100320 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet a76d61c3464b41519d7657796bd853ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a76d61c3464b41519d7657796bd853ff already in progress: creating tablet
I20251024 19:48:00.079620 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6f376570019346199a3c8c77d879fa1c" } modify_external_catalogs: true
I20251024 19:48:00.079746 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e378095ddecc4dcf801a1568160c34b4" } modify_external_catalogs: true
I20251024 19:48:00.100839 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6f376570019346199a3c8c77d879fa1c" } modify_external_catalogs: true
I20251024 19:48:00.100847 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e378095ddecc4dcf801a1568160c34b4" } modify_external_catalogs: true
I20251024 19:48:00.101423 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 5d129bdc876e425aabcf1c99997e319a (DEFAULT_TABLE table=default.Table_9edfe580f57e43919a206badbc0bbfd6 [id=d28fd80220c64f8496a09f5d4e712874]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.101553 1766 tablet_bootstrap.cc:492] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:00.101955 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d129bdc876e425aabcf1c99997e319a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.104300 1766 tablet_bootstrap.cc:654] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.104367 1614 tablet_service.cc:1505] Processing CreateTablet for tablet c82f60819e7a442bbc0cc04659e14e87 (DEFAULT_TABLE table=default.Table_6f376570019346199a3c8c77d879fa1c [id=c5f7c2469f394bba8b91e2f20170963e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.105739 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 2602335327fc489fa154056c5ef5138f (DEFAULT_TABLE table=default.Table_83162e45d21e4d47a1cb086b20db8ef3 [id=d35a25cbeefd47d39a5cba0bed321da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.106176 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2602335327fc489fa154056c5ef5138f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.110661 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c82f60819e7a442bbc0cc04659e14e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.110788 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 4193750331874d7ba9bbe6279c3a8e37 (DEFAULT_TABLE table=default.Table_646390b8820e4af996e1fa10cf39ac39 [id=3172f9d08e8649c289a02ad1cd61177d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.111362 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e0d4700f7fd849399f95eb388d194091" } modify_external_catalogs: true
I20251024 19:48:00.111454 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c3cbf88afafa464099c69d8e017743e8" } modify_external_catalogs: true
I20251024 19:48:00.111625 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e0d4700f7fd849399f95eb388d194091" } modify_external_catalogs: true
I20251024 19:48:00.111721 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c3cbf88afafa464099c69d8e017743e8" } modify_external_catalogs: true
I20251024 19:48:00.112641 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4193750331874d7ba9bbe6279c3a8e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.114745 1767 tablet_bootstrap.cc:654] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.115485 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 (DEFAULT_TABLE table=default.Table_5006c140e0964b569953c4f03495c697 [id=338beb9c6956484daef970b28d5e4b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.116474 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 (DEFAULT_TABLE table=default.Table_5006c140e0964b569953c4f03495c697 [id=338beb9c6956484daef970b28d5e4b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.119628 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.120580 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.121337 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 4193750331874d7ba9bbe6279c3a8e37 (DEFAULT_TABLE table=default.Table_646390b8820e4af996e1fa10cf39ac39 [id=3172f9d08e8649c289a02ad1cd61177d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.123448 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4193750331874d7ba9bbe6279c3a8e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.124867 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet c82f60819e7a442bbc0cc04659e14e87
I20251024 19:48:00.124899 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 15c08dcf379f4df9a0e461d6dcfeeac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.125334 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 2602335327fc489fa154056c5ef5138f (DEFAULT_TABLE table=default.Table_83162e45d21e4d47a1cb086b20db8ef3 [id=d35a25cbeefd47d39a5cba0bed321da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.128270 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 62667092d7e244cca1020ec220ea9124 (DEFAULT_TABLE table=default.Table_06b07944861041f8a66001a6555d7659 [id=96c4981300dc48e6bf55fc633164ed63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.131161 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b1fa439afe5a48a79557019687319af6
I20251024 19:48:00.138742 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 62667092d7e244cca1020ec220ea9124 (DEFAULT_TABLE table=default.Table_06b07944861041f8a66001a6555d7659 [id=96c4981300dc48e6bf55fc633164ed63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.139272 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62667092d7e244cca1020ec220ea9124. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.151425 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet c82f60819e7a442bbc0cc04659e14e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.155526 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 369cebfda6ee43638b79f8652fa55855
I20251024 19:48:00.155905 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 979dd87a8c66480ea314fcc75fa878c3
I20251024 19:48:00.156594 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3887fc21726846279fecb02672b9bc07
I20251024 19:48:00.160902 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet c82f60819e7a442bbc0cc04659e14e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.161805 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet b1fa439afe5a48a79557019687319af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.163200 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 608696ec90ed45e1ad939ed2bfc88d34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:00.163427 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet c82f60819e7a442bbc0cc04659e14e87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c82f60819e7a442bbc0cc04659e14e87 already in progress: creating tablet
I20251024 19:48:00.163659 1775 tablet_replica.cc:333] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:00.163897 1775 raft_consensus.cc:2243] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:00.163970 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b1fa439afe5a48a79557019687319af6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1fa439afe5a48a79557019687319af6 already in progress: creating tablet
I20251024 19:48:00.164099 1775 raft_consensus.cc:2272] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:00.164968 1775 ts_tablet_manager.cc:1916] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.165028 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 067631b128fd4a1e9ab72c9507dab6c4 (DEFAULT_TABLE table=default.Table_719145aefd754122ba1bb0476ca4ac7d [id=3f394c59d82b44928beb22f42312a4a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.165519 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 067631b128fd4a1e9ab72c9507dab6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.171291 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 6f630fd412ca4c4f8570359c1678e6af (DEFAULT_TABLE table=default.Table_a5d11f93359546078ef5becf10539131 [id=e30a8747e9a6453e9567626b55ccfc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.172384 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 6f89a3bbeb9b4d57a49078729110d951 (DEFAULT_TABLE table=default.Table_5a60ae2f44c2462086e73f80ad013457 [id=c469aebfcbde437abe1636f85c39d6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.174197 1774 tablet_replica.cc:333] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:00.172889 1775 ts_tablet_manager.cc:1929] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:00.175071 1775 log.cc:1199] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/608696ec90ed45e1ad939ed2bfc88d34
I20251024 19:48:00.175591 1775 ts_tablet_manager.cc:1950] T 608696ec90ed45e1ad939ed2bfc88d34 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:00.176638 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 369cebfda6ee43638b79f8652fa55855 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.173532 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 3324755a5a184a609fae5f7a25d58e4c (DEFAULT_TABLE table=default.Table_2f543df6b1194a7195ecd656134e4b17 [id=e908b8ad13614c3c90e0f8aaafd0421f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.175537 1479 tablet_service.cc:1505] Processing CreateTablet for tablet c5cbce9e4b61451ba7f458df922be759 (DEFAULT_TABLE table=default.Table_f57b8c8910c2489a80979e2ffe453f3c [id=1817b735a20f46409680f85379280ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.175930 1488 tablet_service.cc:1505] Processing CreateTablet for tablet d6de2c21796f4bc4904f88b654c3baec (DEFAULT_TABLE table=default.Table_894d6739340148a995dd849a205c19c3 [id=18ec057e920e4c21ae8cbe298058b6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.178247 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 979dd87a8c66480ea314fcc75fa878c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.174077 1491 tablet_service.cc:1505] Processing CreateTablet for tablet b240965605904533bd2af99f61aaef26 (DEFAULT_TABLE table=default.Table_a1366192b3134e3abfc231729cb42344 [id=ae6b320a09934ac5873d3b35c33f2c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.174093 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f630fd412ca4c4f8570359c1678e6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.179212 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 3887fc21726846279fecb02672b9bc07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.179443 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5cbce9e4b61451ba7f458df922be759. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.179736 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6de2c21796f4bc4904f88b654c3baec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.179812 1612 tablet_service.cc:1505] Processing CreateTablet for tablet c5cbce9e4b61451ba7f458df922be759 (DEFAULT_TABLE table=default.Table_f57b8c8910c2489a80979e2ffe453f3c [id=1817b735a20f46409680f85379280ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.180536 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 369cebfda6ee43638b79f8652fa55855 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.184773 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3324755a5a184a609fae5f7a25d58e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.186401 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 3324755a5a184a609fae5f7a25d58e4c (DEFAULT_TABLE table=default.Table_2f543df6b1194a7195ecd656134e4b17 [id=e908b8ad13614c3c90e0f8aaafd0421f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.189133 1624 tablet_service.cc:1505] Processing CreateTablet for tablet d6de2c21796f4bc4904f88b654c3baec (DEFAULT_TABLE table=default.Table_894d6739340148a995dd849a205c19c3 [id=18ec057e920e4c21ae8cbe298058b6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.189087 1625 tablet_service.cc:1505] Processing CreateTablet for tablet b240965605904533bd2af99f61aaef26 (DEFAULT_TABLE table=default.Table_a1366192b3134e3abfc231729cb42344 [id=ae6b320a09934ac5873d3b35c33f2c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.191282 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2602335327fc489fa154056c5ef5138f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.191951 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6de2c21796f4bc4904f88b654c3baec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.192267 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b240965605904533bd2af99f61aaef26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.193107 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5cbce9e4b61451ba7f458df922be759. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.193732 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b240965605904533bd2af99f61aaef26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.194931 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3324755a5a184a609fae5f7a25d58e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.195008 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f89a3bbeb9b4d57a49078729110d951. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.204387 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_edfd7941851340ecbcd71d4b48ff583f" } new_table_name: "default.Table_a83dd919e9ab42179bff96aac9a09cec" modify_external_catalogs: true
I20251024 19:48:00.205916 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_823526f1c3164a3ba70c44dd8388fb9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:00.208038 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 639836c656f44a819036f4d00e055bd9 (DEFAULT_TABLE table=default.Table_6eb10ae51f3a43c2ab15f362708fba94 [id=f3080df28ee94876bb6b790cea45a4e8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:00.208628 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_823526f1c3164a3ba70c44dd8388fb9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.208901 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 5d129bdc876e425aabcf1c99997e319a (DEFAULT_TABLE table=default.Table_9edfe580f57e43919a206badbc0bbfd6 [id=d28fd80220c64f8496a09f5d4e712874]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.210638 1882 raft_consensus.cc:493] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:00.210968 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 9489a01434a24d9ca94791252950725c (DEFAULT_TABLE table=default.Table_1c89d244042a4852a9cb4113f6726332 [id=ba207dd9990d4a0e85eaefbb4908dd02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.211309 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639836c656f44a819036f4d00e055bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.212464 1882 raft_consensus.cc:515] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:00.213662 1348 tablet_service.cc:1505] Processing CreateTablet for tablet f83b9290f00145a780318a43249eec12 (DEFAULT_TABLE table=default.Table_dd9b9ef96622439ea016c3813fcbb81a [id=be1cc9d047554ea6a9afa531fb08a81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.214088 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f83b9290f00145a780318a43249eec12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.215903 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e956ea914c545d79be6d632bfc4ec57" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:00.216773 1308 leader_election.cc:343] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:00.217484 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet b1fa439afe5a48a79557019687319af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:00.217747 1343 tablet_service.cc:1505] Processing CreateTablet for tablet c82f60819e7a442bbc0cc04659e14e87 (DEFAULT_TABLE table=default.Table_6f376570019346199a3c8c77d879fa1c [id=c5f7c2469f394bba8b91e2f20170963e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.217870 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 4193750331874d7ba9bbe6279c3a8e37 (DEFAULT_TABLE table=default.Table_646390b8820e4af996e1fa10cf39ac39 [id=3172f9d08e8649c289a02ad1cd61177d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.218091 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c82f60819e7a442bbc0cc04659e14e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.218130 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet c82f60819e7a442bbc0cc04659e14e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:00.218464 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 067631b128fd4a1e9ab72c9507dab6c4 (DEFAULT_TABLE table=default.Table_719145aefd754122ba1bb0476ca4ac7d [id=3f394c59d82b44928beb22f42312a4a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.218848 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 067631b128fd4a1e9ab72c9507dab6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.219827 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 (DEFAULT_TABLE table=default.Table_5006c140e0964b569953c4f03495c697 [id=338beb9c6956484daef970b28d5e4b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.220177 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.220394 1882 leader_election.cc:290] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:00.220491 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 62667092d7e244cca1020ec220ea9124 (DEFAULT_TABLE table=default.Table_06b07944861041f8a66001a6555d7659 [id=96c4981300dc48e6bf55fc633164ed63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.220816 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4193750331874d7ba9bbe6279c3a8e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.220854 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 2602335327fc489fa154056c5ef5138f (DEFAULT_TABLE table=default.Table_83162e45d21e4d47a1cb086b20db8ef3 [id=d35a25cbeefd47d39a5cba0bed321da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.221189 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2602335327fc489fa154056c5ef5138f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.224875 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62667092d7e244cca1020ec220ea9124. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.225703 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9489a01434a24d9ca94791252950725c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.226776 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 6f630fd412ca4c4f8570359c1678e6af (DEFAULT_TABLE table=default.Table_a5d11f93359546078ef5becf10539131 [id=e30a8747e9a6453e9567626b55ccfc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.227342 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f630fd412ca4c4f8570359c1678e6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.229398 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d129bdc876e425aabcf1c99997e319a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.231251 1774 raft_consensus.cc:2243] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:00.231994 1768 tablet_bootstrap.cc:492] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:00.249795 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62667092d7e244cca1020ec220ea9124. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.255486 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3f7f7b5dfa674b19ad19c339c619326c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.256476 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f7f7b5dfa674b19ad19c339c619326c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.259069 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 6f89a3bbeb9b4d57a49078729110d951 (DEFAULT_TABLE table=default.Table_5a60ae2f44c2462086e73f80ad013457 [id=c469aebfcbde437abe1636f85c39d6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.259712 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f89a3bbeb9b4d57a49078729110d951. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:00.256914 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 913534ab88dd459e9cac2faebb5492a1 consensus state has no leader
I20251024 19:48:00.273438 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 6f630fd412ca4c4f8570359c1678e6af (DEFAULT_TABLE table=default.Table_a5d11f93359546078ef5becf10539131 [id=e30a8747e9a6453e9567626b55ccfc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.274412 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f630fd412ca4c4f8570359c1678e6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.277251 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 3887fc21726846279fecb02672b9bc07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.277817 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet b1fa439afe5a48a79557019687319af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.278388 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 067631b128fd4a1e9ab72c9507dab6c4 (DEFAULT_TABLE table=default.Table_719145aefd754122ba1bb0476ca4ac7d [id=3f394c59d82b44928beb22f42312a4a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.278977 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 067631b128fd4a1e9ab72c9507dab6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.285564 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d5ca9b9774254589b656ade4f08dc4c1" } new_table_name: "default.Table_3819751f36bf419e9edd0f1ff339713a" modify_external_catalogs: true
I20251024 19:48:00.288391 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d9b6a6d63d7b424991e461c8b6065823" } new_table_name: "default.Table_b04513e51b4647aebeee3f6e97e6c8e6" modify_external_catalogs: true
I20251024 19:48:00.290277 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 844638a6b6944d0394df3e78a8ea51d6 (DEFAULT_TABLE table=default.Table_823526f1c3164a3ba70c44dd8388fb9c [id=fc5fe814531e4110ace3fb312aefa7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.290701 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 844638a6b6944d0394df3e78a8ea51d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.290577 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d8ae96b95894421588b9b807f296a627" } new_table_name: "default.Table_73923b0648674b2891cbbf4b90d3f106" modify_external_catalogs: true
I20251024 19:48:00.290627 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_023e47eac6c24e328d5a01fcf3b2f579"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:00.291028 1766 tablet_bootstrap.cc:492] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:00.291185 1766 ts_tablet_manager.cc:1403] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.190s user 0.010s sys 0.048s
W20251024 19:48:00.291246 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_023e47eac6c24e328d5a01fcf3b2f579 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.291983 1766 raft_consensus.cc:359] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:00.292162 1766 raft_consensus.cc:385] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.292243 1766 raft_consensus.cc:740] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.292515 1766 consensus_queue.cc:260] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:00.293192 1766 ts_tablet_manager.cc:1434] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:00.293469 1766 tablet_bootstrap.cc:492] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:00.295487 1766 tablet_bootstrap.cc:654] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.298528 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3bd0db5c2861405a9dd9d66731cd9a57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:00.303984 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 979dd87a8c66480ea314fcc75fa878c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.304322 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c777799df90e491992909e8a45205a06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.304929 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c777799df90e491992909e8a45205a06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.305397 1776 ts_tablet_manager.cc:1929] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251024 19:48:00.305701 1776 log.cc:1199] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/f1ad65fd080e452e8094975d420336c7
I20251024 19:48:00.306489 1776 ts_tablet_manager.cc:1950] T f1ad65fd080e452e8094975d420336c7 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:00.308996 1776 tablet_replica.cc:333] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:00.309338 1776 raft_consensus.cc:2243] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:00.310563 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 3324755a5a184a609fae5f7a25d58e4c (DEFAULT_TABLE table=default.Table_2f543df6b1194a7195ecd656134e4b17 [id=e908b8ad13614c3c90e0f8aaafd0421f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.311033 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3324755a5a184a609fae5f7a25d58e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.311431 1768 ts_tablet_manager.cc:1403] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.216s user 0.005s sys 0.008s
W20251024 19:48:00.312229 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bd0db5c2861405a9dd9d66731cd9a57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.312413 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet f1ad65fd080e452e8094975d420336c7 (table default.Table_a54ffafa18b848b79a290f58c9754bec [id=2fd5c541b06a46469e65ed0942d2c394]) successfully deleted
W20251024 19:48:00.312708 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 27b32126d04e480f8db7b2b20aba5873 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27b32126d04e480f8db7b2b20aba5873 already in progress: creating tablet
W20251024 19:48:00.312943 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2037fcca87b94b5fb31639b4d07307b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2037fcca87b94b5fb31639b4d07307b0 already in progress: creating tablet
W20251024 19:48:00.313143 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet fca73ab61f0c47df9186459caf2b9a32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fca73ab61f0c47df9186459caf2b9a32 already in progress: creating tablet
W20251024 19:48:00.313339 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 44ee03ede02c4bdab706fde910978332 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ee03ede02c4bdab706fde910978332 already in progress: creating tablet
I20251024 19:48:00.314756 1882 raft_consensus.cc:493] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:00.315532 1774 raft_consensus.cc:2272] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:00.315346 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 6f89a3bbeb9b4d57a49078729110d951 (DEFAULT_TABLE table=default.Table_5a60ae2f44c2462086e73f80ad013457 [id=c469aebfcbde437abe1636f85c39d6fb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:00.327529 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fdfa5eea3b674a3389a156d1c72204df consensus state has no leader
I20251024 19:48:00.327620 1350 tablet_service.cc:1505] Processing CreateTablet for tablet b240965605904533bd2af99f61aaef26 (DEFAULT_TABLE table=default.Table_a1366192b3134e3abfc231729cb42344 [id=ae6b320a09934ac5873d3b35c33f2c65]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:00.332636 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07e75c7f35544bec82cb994dea5ee4ef consensus state has no leader
I20251024 19:48:00.332564 1349 tablet_service.cc:1505] Processing CreateTablet for tablet d6de2c21796f4bc4904f88b654c3baec (DEFAULT_TABLE table=default.Table_894d6739340148a995dd849a205c19c3 [id=18ec057e920e4c21ae8cbe298058b6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.333812 1882 raft_consensus.cc:515] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.338563 1767 tablet_bootstrap.cc:492] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:00.338914 1767 ts_tablet_manager.cc:1403] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.245s user 0.025s sys 0.047s
I20251024 19:48:00.340166 1767 raft_consensus.cc:359] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.340497 1767 raft_consensus.cc:385] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.340636 1767 raft_consensus.cc:740] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.341022 1767 consensus_queue.cc:260] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.343448 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f89a3bbeb9b4d57a49078729110d951. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.343461 1774 ts_tablet_manager.cc:1916] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.345351 1767 ts_tablet_manager.cc:1434] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251024 19:48:00.346288 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet 110907ae943f45b5bcaa9b66ccdb84fd with delete_type TABLET_DATA_DELETED (replaced by tablet 1d04a9b54bd6446d8deb64292cef672d) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.347018 1767 tablet_bootstrap.cc:492] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:00.347702 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e956ea914c545d79be6d632bfc4ec57" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:00.347982 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8721ee81e4e24153a0aad8ce99f872e1" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:00.350939 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8721ee81e4e24153a0aad8ce99f872e1" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:00.351632 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 369cebfda6ee43638b79f8652fa55855 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:00.353305 1308 leader_election.cc:343] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:00.353582 1882 leader_election.cc:290] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:00.354265 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 3887fc21726846279fecb02672b9bc07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:00.355813 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6de2c21796f4bc4904f88b654c3baec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.358789 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet 979dd87a8c66480ea314fcc75fa878c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:00.362392 1776 raft_consensus.cc:2272] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:00.363951 1776 ts_tablet_manager.cc:1916] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.367937 1348 tablet_service.cc:1505] Processing CreateTablet for tablet c5cbce9e4b61451ba7f458df922be759 (DEFAULT_TABLE table=default.Table_f57b8c8910c2489a80979e2ffe453f3c [id=1817b735a20f46409680f85379280ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.369431 1767 tablet_bootstrap.cc:654] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.369700 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b240965605904533bd2af99f61aaef26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.370070 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5cbce9e4b61451ba7f458df922be759. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:00.370397 1305 leader_election.cc:343] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:00.374241 1305 leader_election.cc:304] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
W20251024 19:48:00.375650 1305 leader_election.cc:343] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:00.375851 1305 leader_election.cc:304] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:00.376135 2026 raft_consensus.cc:2749] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:00.376430 1305 consensus_peers.cc:597] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97c6dce647064344b0b53a7ffaeafb75. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:00.376785 1882 raft_consensus.cc:2749] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:00.377050 1825 rpc_server.cc:307] RPC server started. Bound to: 127.0.252.254:36505
I20251024 19:48:00.377638 1768 raft_consensus.cc:359] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.377868 1768 raft_consensus.cc:385] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.381081 1825 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data/info.pb
I20251024 19:48:00.382735 2032 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.252.254:36505 every 8 connection(s)
I20251024 19:48:00.390085 1011 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskc9NInd/build/asan/bin/kudu as pid 1825
I20251024 19:48:00.397850 1887 raft_consensus.cc:493] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:00.398089 1887 raft_consensus.cc:515] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.398322 1768 raft_consensus.cc:740] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.398922 1887 leader_election.cc:290] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:00.400646 1308 consensus_peers.cc:597] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:00.401163 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15c08dcf379f4df9a0e461d6dcfeeac4" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:00.401949 1586 leader_election.cc:343] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:00.402161 1586 leader_election.cc:343] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 15c08dcf379f4df9a0e461d6dcfeeac4
I20251024 19:48:00.402287 1586 leader_election.cc:304] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:00.402798 1887 raft_consensus.cc:2749] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:00.407227 1768 consensus_queue.cc:260] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.409040 1768 ts_tablet_manager.cc:1434] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.097s user 0.002s sys 0.001s
W20251024 19:48:00.413866 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3259823b622461fa823841925b3d203 consensus state has no leader
I20251024 19:48:00.416039 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_23a3d28e6ce941008c62ffbf100cc922" } new_table_name: "default.Table_55f8c66bd97343438051a98af12d8c2c" modify_external_catalogs: true
I20251024 19:48:00.425504 1776 ts_tablet_manager.cc:1929] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:00.426017 1768 tablet_bootstrap.cc:492] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:00.426235 1776 log.cc:1199] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/fb17a35d8d8446cf9870d05e107ed237
I20251024 19:48:00.426843 1776 ts_tablet_manager.cc:1950] T fb17a35d8d8446cf9870d05e107ed237 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:00.429852 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_bbd727eca62c408ca76741ebea86eb3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.430395 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbd727eca62c408ca76741ebea86eb3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.433280 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_25adc9559f2b43e2a4adc92dfc0c7807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.433884 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25adc9559f2b43e2a4adc92dfc0c7807 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:00.434015 1308 consensus_peers.cc:597] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97c6dce647064344b0b53a7ffaeafb75. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:00.434489 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet fb17a35d8d8446cf9870d05e107ed237 (table default.Table_51982d999b0847ec87fc62a473782a5c [id=9a05bcf7ad5d430c9ef6d6e18a27808e]) successfully deleted
W20251024 19:48:00.435393 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet a76d61c3464b41519d7657796bd853ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a76d61c3464b41519d7657796bd853ff already in progress: creating tablet
W20251024 19:48:00.435976 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 0a12e415590a4c71aab4ad7d30412c83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a12e415590a4c71aab4ad7d30412c83 already in progress: creating tablet
I20251024 19:48:00.436587 1776 tablet_replica.cc:333] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:00.440825 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_da1c2655e0de479e91cf0a21e1a728b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.441398 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da1c2655e0de479e91cf0a21e1a728b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.441807 1776 raft_consensus.cc:2243] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:00.441998 1776 raft_consensus.cc:2272] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:00.446440 1776 ts_tablet_manager.cc:1916] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.449122 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet 50d919a0b3d74e9a97d136a84d50e24b from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:00.449977 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet 90405a0c13bd4c229b011c90232f751d from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:00.450861 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7b4b72b748a04135803b67edd2b45447"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.451762 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b4b72b748a04135803b67edd2b45447 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.457268 1774 ts_tablet_manager.cc:1929] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:00.462289 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_740023c3917c4850b564d096c64670d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.463109 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_740023c3917c4850b564d096c64670d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.465314 2026 raft_consensus.cc:493] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:00.467041 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_71bcb15d38b842efbdfc0fb90171e71e" } modify_external_catalogs: true
I20251024 19:48:00.467303 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_71bcb15d38b842efbdfc0fb90171e71e" } modify_external_catalogs: true
I20251024 19:48:00.467823 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_acb41c25b40742ef92f4bb9d45ada0a3" } modify_external_catalogs: true
I20251024 19:48:00.467999 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_acb41c25b40742ef92f4bb9d45ada0a3" } modify_external_catalogs: true
I20251024 19:48:00.468091 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5006c140e0964b569953c4f03495c697" } modify_external_catalogs: true
I20251024 19:48:00.468248 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5006c140e0964b569953c4f03495c697" } modify_external_catalogs: true
I20251024 19:48:00.467864 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_69e288a9d6d24a5f841f62cd9a208f0f" } modify_external_catalogs: true
I20251024 19:48:00.468590 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_69e288a9d6d24a5f841f62cd9a208f0f" } modify_external_catalogs: true
I20251024 19:48:00.469282 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 844638a6b6944d0394df3e78a8ea51d6 (DEFAULT_TABLE table=default.Table_823526f1c3164a3ba70c44dd8388fb9c [id=fc5fe814531e4110ace3fb312aefa7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.469893 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 844638a6b6944d0394df3e78a8ea51d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.471727 2026 raft_consensus.cc:515] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.472436 2026 leader_election.cc:290] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:00.473222 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f91316c030dc4f7da27977a9dde01f4b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:00.473569 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f91316c030dc4f7da27977a9dde01f4b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:00.473834 1305 leader_election.cc:343] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:00.476644 1768 tablet_bootstrap.cc:654] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.476938 1774 log.cc:1199] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/9820e68ad5264df1a79b22f3b5369c5a
W20251024 19:48:00.476644 1308 leader_election.cc:343] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:00.477494 1308 leader_election.cc:304] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:00.478119 2026 raft_consensus.cc:2749] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:00.478781 1776 ts_tablet_manager.cc:1929] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:00.479100 1887 raft_consensus.cc:493] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:00.481523 1887 raft_consensus.cc:515] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.482332 1887 leader_election.cc:290] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:00.483124 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ba85164015448d4a688a473cce90c5c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:00.483359 1371 raft_consensus.cc:2468] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:00.483948 1586 leader_election.cc:304] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:00.484383 1887 raft_consensus.cc:2804] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:00.484553 1887 raft_consensus.cc:493] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:00.484652 1887 raft_consensus.cc:3060] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:00.485072 1767 tablet_bootstrap.cc:492] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:00.485256 1767 ts_tablet_manager.cc:1403] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.138s user 0.002s sys 0.005s
I20251024 19:48:00.486181 1767 raft_consensus.cc:359] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.486523 1767 raft_consensus.cc:385] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.486745 1767 raft_consensus.cc:740] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.490132 1774 ts_tablet_manager.cc:1950] T 9820e68ad5264df1a79b22f3b5369c5a P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:00.491530 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 132) took 2691 ms. Trace:
I20251024 19:48:00.491679 1774 rpcz_store.cc:276] 1024 19:47:57.800112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.800322 (+ 210us) service_pool.cc:225] Handling call
1024 19:48:00.173047 (+2372725us) ts_tablet_manager.cc:1151] Deleting tablet 9820e68ad5264df1a79b22f3b5369c5a
1024 19:48:00.173063 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:00.445474 (+272411us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.457164 (+ 11690us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.491479 (+ 34315us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2372503}
I20251024 19:48:00.493105 1774 tablet_replica.cc:333] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:00.493352 1774 raft_consensus.cc:2243] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
W20251024 19:48:00.494174 1177 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet a3259823b622461fa823841925b3d203 (table default.Table_3819751f36bf419e9edd0f1ff339713a [id=350b37bf42194fcfbb660bffe83e47c4], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet a3259823b622461fa823841925b3d203 consensus state has no leader
I20251024 19:48:00.495231 1774 pending_rounds.cc:70] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:00.495322 1774 pending_rounds.cc:77] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429303636738048 op_type: NO_OP noop_request { }
I20251024 19:48:00.495357 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 844638a6b6944d0394df3e78a8ea51d6 (DEFAULT_TABLE table=default.Table_823526f1c3164a3ba70c44dd8388fb9c [id=fc5fe814531e4110ace3fb312aefa7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.495501 1774 raft_consensus.cc:2889] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:00.495630 1774 raft_consensus.cc:2272] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:00.495973 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 844638a6b6944d0394df3e78a8ea51d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.496299 1774 ts_tablet_manager.cc:1916] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.510118 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ba85164015448d4a688a473cce90c5c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:00.510329 1767 consensus_queue.cc:260] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.512343 1776 log.cc:1199] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/3caecdff61ce48f7b7c3846bb380d21e
I20251024 19:48:00.512828 1776 ts_tablet_manager.cc:1950] T 3caecdff61ce48f7b7c3846bb380d21e P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:00.514058 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 3caecdff61ce48f7b7c3846bb380d21e (table default.Table_8c169b038d764be684a4e5b56faf745b [id=e3c15f9790304d4eb7be44889ef89d20]) successfully deleted
I20251024 19:48:00.514317 1776 tablet_replica.cc:333] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:00.514509 1776 raft_consensus.cc:2243] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:00.514765 1776 raft_consensus.cc:2272] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:00.517009 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ade7993306ee438baebbf639e2109768" } modify_external_catalogs: true
I20251024 19:48:00.517236 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ade7993306ee438baebbf639e2109768" } modify_external_catalogs: true
I20251024 19:48:00.521647 1887 raft_consensus.cc:515] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.522372 1887 leader_election.cc:290] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:00.522526 1768 tablet_bootstrap.cc:492] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:00.524585 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ba85164015448d4a688a473cce90c5c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:00.524845 1371 raft_consensus.cc:3060] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:00.525146 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 608696ec90ed45e1ad939ed2bfc88d34 (table default.Table_6e1fbc007b464e8c93803ba262a19fea [id=2c92cb09b287410c90b811e4b88c2260]) successfully deleted
W20251024 19:48:00.525612 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 369cebfda6ee43638b79f8652fa55855 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 369cebfda6ee43638b79f8652fa55855 already in progress: creating tablet
W20251024 19:48:00.525885 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 979dd87a8c66480ea314fcc75fa878c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 979dd87a8c66480ea314fcc75fa878c3 already in progress: creating tablet
W20251024 19:48:00.526077 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3887fc21726846279fecb02672b9bc07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3887fc21726846279fecb02672b9bc07 already in progress: creating tablet
I20251024 19:48:00.527303 1776 ts_tablet_manager.cc:1916] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.527702 1371 raft_consensus.cc:2468] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:00.528270 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet cc0bf7aee1a1421b9d985020a1674639 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.528899 1768 ts_tablet_manager.cc:1403] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.110s user 0.008s sys 0.002s
I20251024 19:48:00.529029 1586 leader_election.cc:304] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:00.529518 1887 raft_consensus.cc:2804] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:00.529765 1887 raft_consensus.cc:697] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:00.529883 1768 raft_consensus.cc:359] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.530093 1768 raft_consensus.cc:385] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.530170 1768 raft_consensus.cc:740] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.530243 1887 consensus_queue.cc:237] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.530580 1768 consensus_queue.cc:260] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:00.532104 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 187925082cdd4923a6e9148bf9f270fd consensus state has no leader
I20251024 19:48:00.535873 1768 ts_tablet_manager.cc:1434] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251024 19:48:00.536340 1768 tablet_bootstrap.cc:492] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:00.541070 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet f1ad65fd080e452e8094975d420336c7 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:00.541472 1776 ts_tablet_manager.cc:1929] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:00.541651 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:00.544363 2034 sys_catalog.cc:263] Verifying existing consensus state
I20251024 19:48:00.545899 1768 tablet_bootstrap.cc:654] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.556234 1776 log.cc:1199] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/15c08dcf379f4df9a0e461d6dcfeeac4
I20251024 19:48:00.556283 1774 ts_tablet_manager.cc:1929] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:00.556458 1774 log.cc:1199] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/65f646bd05274cb6859c0b7f92b61352
I20251024 19:48:00.556969 1774 ts_tablet_manager.cc:1950] T 65f646bd05274cb6859c0b7f92b61352 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:00.557017 1776 ts_tablet_manager.cc:1950] T 15c08dcf379f4df9a0e461d6dcfeeac4 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:00.559996 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ba85164015448d4a688a473cce90c5c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:00.560549 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d941eafb1414f068dc656946082f617 with delete_type TABLET_DATA_DELETED (replaced by tablet f91316c030dc4f7da27977a9dde01f4b) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.560846 1775 tablet_replica.cc:333] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:00.561045 1775 raft_consensus.cc:2243] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:00.561199 1775 raft_consensus.cc:2272] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:00.562145 1775 ts_tablet_manager.cc:1916] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.564352 1776 tablet_replica.cc:333] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:00.564597 1776 raft_consensus.cc:2243] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:00.565819 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 15c08dcf379f4df9a0e461d6dcfeeac4 (table default.Table_96c033f8e7ab4c919b38c80fabe3e367 [id=473fc6762db94b959bb9a260b11140f5]) successfully deleted
W20251024 19:48:00.566159 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet c82f60819e7a442bbc0cc04659e14e87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c82f60819e7a442bbc0cc04659e14e87 already in progress: creating tablet
W20251024 19:48:00.566344 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 369cebfda6ee43638b79f8652fa55855 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 369cebfda6ee43638b79f8652fa55855 already in progress: creating tablet
W20251024 19:48:00.566520 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3887fc21726846279fecb02672b9bc07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3887fc21726846279fecb02672b9bc07 already in progress: creating tablet
W20251024 19:48:00.566661 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b1fa439afe5a48a79557019687319af6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1fa439afe5a48a79557019687319af6 already in progress: creating tablet
W20251024 19:48:00.566854 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 979dd87a8c66480ea314fcc75fa878c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 979dd87a8c66480ea314fcc75fa878c3 already in progress: creating tablet
I20251024 19:48:00.567056 1776 raft_consensus.cc:2272] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:00.559567 1586 leader_election.cc:343] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:00.567718 1586 leader_election.cc:343] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:00.568184 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet fe97e31310ae4733ba93de1b62a37842 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:00.568351 1776 ts_tablet_manager.cc:1916] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.568622 1768 tablet_bootstrap.cc:492] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:00.568841 1768 ts_tablet_manager.cc:1403] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20251024 19:48:00.569763 1768 raft_consensus.cc:359] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.570012 1768 raft_consensus.cc:385] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.570135 1768 raft_consensus.cc:740] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.570447 1768 consensus_queue.cc:260] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.572285 1768 ts_tablet_manager.cc:1434] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:00.572618 1768 tablet_bootstrap.cc:492] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:00.573187 1775 ts_tablet_manager.cc:1929] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:00.573380 1775 log.cc:1199] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/2d941eafb1414f068dc656946082f617
I20251024 19:48:00.573827 1775 ts_tablet_manager.cc:1950] T 2d941eafb1414f068dc656946082f617 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:00.574921 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 2d941eafb1414f068dc656946082f617 (table default.Table_cdf2e8872b724c7aa86d5091eb52fb1c [id=a402387542654ccf8e0b04ca8ecebb1a]) successfully deleted
I20251024 19:48:00.576865 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d941eafb1414f068dc656946082f617 with delete_type TABLET_DATA_DELETED (replaced by tablet f91316c030dc4f7da27977a9dde01f4b) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:00.577427 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2d941eafb1414f068dc656946082f617 because the tablet was not found. No further retry: Not found: Tablet not found: 2d941eafb1414f068dc656946082f617
I20251024 19:48:00.578490 1766 tablet_bootstrap.cc:492] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:00.578560 1768 tablet_bootstrap.cc:654] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.579011 1766 ts_tablet_manager.cc:1403] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.286s user 0.045s sys 0.083s
I20251024 19:48:00.579907 2048 raft_consensus.cc:493] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:00.579888 1766 raft_consensus.cc:359] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.580160 1766 raft_consensus.cc:385] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.580117 2048 raft_consensus.cc:515] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.580282 1766 raft_consensus.cc:740] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.580617 1766 consensus_queue.cc:260] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.581326 1766 ts_tablet_manager.cc:1434] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:00.581794 1766 tablet_bootstrap.cc:492] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:00.583686 1766 tablet_bootstrap.cc:654] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.579447 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 128) took 2790 ms. Trace:
I20251024 19:48:00.588138 1774 rpcz_store.cc:276] 1024 19:47:57.789223 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.797332 (+ 8109us) service_pool.cc:225] Handling call
1024 19:48:00.493008 (+2695676us) ts_tablet_manager.cc:1151] Deleting tablet 65f646bd05274cb6859c0b7f92b61352
1024 19:48:00.493020 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:00.549841 (+ 56821us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.556131 (+ 6290us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.579388 (+ 23257us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2682082}
I20251024 19:48:00.589893 1774 tablet_replica.cc:333] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:00.590126 1774 raft_consensus.cc:2243] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:00.590377 1774 raft_consensus.cc:2272] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:00.590912 1774 ts_tablet_manager.cc:1916] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.592952 1766 tablet_bootstrap.cc:492] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:00.593524 1766 ts_tablet_manager.cc:1403] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20251024 19:48:00.593995 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc0bf7aee1a1421b9d985020a1674639" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:00.594696 1586 leader_election.cc:343] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:00.595315 1766 raft_consensus.cc:359] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.595666 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 94fc8c5716af47faa4e831f05fce8610 (DEFAULT_TABLE table=default.Table_023e47eac6c24e328d5a01fcf3b2f579 [id=048d560cd32a44c5abf12bde27fe35c7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:00.596144 1586 leader_election.cc:343] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: cc0bf7aee1a1421b9d985020a1674639
I20251024 19:48:00.596267 1586 leader_election.cc:304] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:00.597033 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94fc8c5716af47faa4e831f05fce8610. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.597368 2048 leader_election.cc:290] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:00.598634 2048 raft_consensus.cc:2749] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:00.599201 2026 raft_consensus.cc:493] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:00.599373 2026 raft_consensus.cc:515] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.599879 1186 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 8ba85164015448d4a688a473cce90c5c (table default.Table_9764ce0360fd4fcfb03ab7df33ed112a [id=0fa321e2d4d847e69435f6de429d8779]). Expected version 1 got 0
I20251024 19:48:00.600036 2026 leader_election.cc:290] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:00.600770 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2ea545b2cc14f589eda9419e55234bf" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:00.600998 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2ea545b2cc14f589eda9419e55234bf" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:00.601307 1305 leader_election.cc:343] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:00.601697 1308 leader_election.cc:343] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:00.601874 1308 leader_election.cc:304] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:00.602566 2026 raft_consensus.cc:2749] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:00.603304 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 94fc8c5716af47faa4e831f05fce8610 (DEFAULT_TABLE table=default.Table_023e47eac6c24e328d5a01fcf3b2f579 [id=048d560cd32a44c5abf12bde27fe35c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.607983 1183 catalog_manager.cc:5649] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:00.608789 1183 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 8ba85164015448d4a688a473cce90c5c (table default.Table_9764ce0360fd4fcfb03ab7df33ed112a [id=0fa321e2d4d847e69435f6de429d8779]). Expected version 1 got 0
I20251024 19:48:00.610329 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94fc8c5716af47faa4e831f05fce8610. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.613674 1766 raft_consensus.cc:385] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.613970 1766 raft_consensus.cc:740] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.614357 1766 consensus_queue.cc:260] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.615540 1766 ts_tablet_manager.cc:1434] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20251024 19:48:00.616451 1766 tablet_bootstrap.cc:492] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:00.618042 2034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Bootstrap starting.
I20251024 19:48:00.621968 1766 tablet_bootstrap.cc:654] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.622575 1696 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:00.630265 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7352d84226db467dad31ae5941842b80
I20251024 19:48:00.632558 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 7352d84226db467dad31ae5941842b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:00.634227 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 7352d84226db467dad31ae5941842b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.635632 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f
I20251024 19:48:00.636080 1192 catalog_manager.cc:6719] ReplaceTablet: tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f deleted and replaced by tablet 9a1031d785b646b095ece9be666d2a77
I20251024 19:48:00.636796 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f with delete_type TABLET_DATA_DELETED (replaced by tablet 9a1031d785b646b095ece9be666d2a77) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.637552 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f with delete_type TABLET_DATA_DELETED (replaced by tablet 9a1031d785b646b095ece9be666d2a77) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:00.638684 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7
I20251024 19:48:00.639743 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.640532 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:00.641808 2034 log.cc:826] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Log is configured to *not* fsync() on all Append() calls
I20251024 19:48:00.642109 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet cef80e4d14bb4e719bceebf43041d079
I20251024 19:48:00.643049 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet cef80e4d14bb4e719bceebf43041d079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.643589 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet cef80e4d14bb4e719bceebf43041d079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:00.645149 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet e20881e400cf4fac914b70afc79c084f
I20251024 19:48:00.646189 1776 ts_tablet_manager.cc:1929] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:00.646346 1776 log.cc:1199] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/110907ae943f45b5bcaa9b66ccdb84fd
I20251024 19:48:00.646816 1776 ts_tablet_manager.cc:1950] T 110907ae943f45b5bcaa9b66ccdb84fd P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:00.646991 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet e20881e400cf4fac914b70afc79c084f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:00.647070 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet e20881e400cf4fac914b70afc79c084f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.647687 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 346383e85cc64b3a8da70a71868ceccc
I20251024 19:48:00.649067 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 110907ae943f45b5bcaa9b66ccdb84fd (table default.Table_2cbb0aad326e4bae85d4c5cb833150c6 [id=1acdc5d0bfb94e0faf47055b20d803b0]) successfully deleted
I20251024 19:48:00.649295 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 346383e85cc64b3a8da70a71868ceccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:00.649335 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet cc0bf7aee1a1421b9d985020a1674639 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc0bf7aee1a1421b9d985020a1674639 already in progress: creating tablet
W20251024 19:48:00.649621 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7352d84226db467dad31ae5941842b80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7352d84226db467dad31ae5941842b80 already in progress: creating tablet
I20251024 19:48:00.650830 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 346383e85cc64b3a8da70a71868ceccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:00.651273 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 346383e85cc64b3a8da70a71868ceccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.632000 1557 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:00.633344 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 7352d84226db467dad31ae5941842b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.640187 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.644011 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet cef80e4d14bb4e719bceebf43041d079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.646610 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet e20881e400cf4fac914b70afc79c084f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.653949 1418 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:00.662325 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet fe97e31310ae4733ba93de1b62a37842
I20251024 19:48:00.662873 1184 catalog_manager.cc:6719] ReplaceTablet: tablet fe97e31310ae4733ba93de1b62a37842 deleted and replaced by tablet 3e6db58fac5e4af9b27bde6c223aa845
I20251024 19:48:00.664469 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet fe97e31310ae4733ba93de1b62a37842 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e6db58fac5e4af9b27bde6c223aa845) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:00.664705 1774 ts_tablet_manager.cc:1929] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:00.665094 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet fe97e31310ae4733ba93de1b62a37842 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e6db58fac5e4af9b27bde6c223aa845) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.665160 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet fe97e31310ae4733ba93de1b62a37842 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e6db58fac5e4af9b27bde6c223aa845) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.665355 1774 log.cc:1199] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/3caecdff61ce48f7b7c3846bb380d21e
I20251024 19:48:00.665889 1766 tablet_bootstrap.cc:492] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:00.666150 1766 ts_tablet_manager.cc:1403] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.001s
I20251024 19:48:00.666962 1766 raft_consensus.cc:359] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.669203 1766 raft_consensus.cc:385] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.669246 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 65f646bd05274cb6859c0b7f92b61352 with delete_type TABLET_DATA_DELETED (replaced by tablet aebf76d4a66947ed9756643740418798) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.669368 1766 raft_consensus.cc:740] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.668294 1774 ts_tablet_manager.cc:1950] T 3caecdff61ce48f7b7c3846bb380d21e P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:48:00.670913 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f already in progress: creating tablet
I20251024 19:48:00.671072 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_605117e2eb734e6b8dc34ceb4b5b5b71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.671725 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_605117e2eb734e6b8dc34ceb4b5b5b71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.671789 1766 consensus_queue.cc:260] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:00.672130 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 already in progress: creating tablet
I20251024 19:48:00.672627 1766 ts_tablet_manager.cc:1434] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251024 19:48:00.673157 1766 tablet_bootstrap.cc:492] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:00.673197 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet cef80e4d14bb4e719bceebf43041d079 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cef80e4d14bb4e719bceebf43041d079 already in progress: creating tablet
W20251024 19:48:00.674440 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet e20881e400cf4fac914b70afc79c084f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e20881e400cf4fac914b70afc79c084f already in progress: creating tablet
W20251024 19:48:00.675483 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 346383e85cc64b3a8da70a71868ceccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 346383e85cc64b3a8da70a71868ceccc already in progress: creating tablet
W20251024 19:48:00.676589 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet fe97e31310ae4733ba93de1b62a37842 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe97e31310ae4733ba93de1b62a37842 already in progress: creating tablet
I20251024 19:48:00.676717 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d1b030ba4f784a2b80386b386edec4d9" } new_table_name: "default.Table_55339e02660843c3a8536a4b7a06831e" modify_external_catalogs: true
I20251024 19:48:00.677847 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 134) took 2855 ms. Trace:
I20251024 19:48:00.678855 2020 raft_consensus.cc:493] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:00.679025 2020 raft_consensus.cc:515] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.679834 2020 leader_election.cc:290] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:00.678823 1774 rpcz_store.cc:276] 1024 19:47:57.821980 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.822043 (+ 63us) service_pool.cc:225] Handling call
1024 19:48:00.589804 (+2767761us) ts_tablet_manager.cc:1151] Deleting tablet 3caecdff61ce48f7b7c3846bb380d21e
1024 19:48:00.589815 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:00.637532 (+ 47717us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.663834 (+ 26302us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.677700 (+ 13866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2764268}
I20251024 19:48:00.680549 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 3caecdff61ce48f7b7c3846bb380d21e (table default.Table_8c169b038d764be684a4e5b56faf745b [id=e3c15f9790304d4eb7be44889ef89d20]) successfully deleted
I20251024 19:48:00.680539 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ba85164015448d4a688a473cce90c5c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:00.684657 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ba85164015448d4a688a473cce90c5c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:00.685040 1371 raft_consensus.cc:2393] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10a9536207e343f188e0ef709081d04e in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:00.689153 1775 tablet_replica.cc:333] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:00.689162 1766 tablet_bootstrap.cc:654] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.689371 1775 raft_consensus.cc:2243] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:00.689631 1775 raft_consensus.cc:2272] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:00.690518 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 346383e85cc64b3a8da70a71868ceccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 346383e85cc64b3a8da70a71868ceccc already in progress: creating tablet
W20251024 19:48:00.690822 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7352d84226db467dad31ae5941842b80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7352d84226db467dad31ae5941842b80 already in progress: creating tablet
W20251024 19:48:00.691066 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 already in progress: creating tablet
W20251024 19:48:00.691309 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet cef80e4d14bb4e719bceebf43041d079 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cef80e4d14bb4e719bceebf43041d079 already in progress: creating tablet
W20251024 19:48:00.691550 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet e20881e400cf4fac914b70afc79c084f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e20881e400cf4fac914b70afc79c084f already in progress: creating tablet
W20251024 19:48:00.691798 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet fe97e31310ae4733ba93de1b62a37842 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe97e31310ae4733ba93de1b62a37842 already in progress: creating tablet
I20251024 19:48:00.681807 1774 tablet_replica.cc:333] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:00.693120 1447 leader_election.cc:304] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:00.693516 2020 raft_consensus.cc:3060] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:00.693894 1774 raft_consensus.cc:2243] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:00.696082 1774 raft_consensus.cc:2272] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:00.697216 2020 raft_consensus.cc:2749] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:00.697877 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b15672d0c13d4c939f61e30a2feb96e6" } new_table_name: "default.Table_c570270b7d7b4f5091c34e0550dd46ef" modify_external_catalogs: true
I20251024 19:48:00.699620 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c10fcb6694c142e8b806f745a3ddc82e" } new_table_name: "default.Table_773b5ef92fac472a8393530fed39600a" modify_external_catalogs: true
I20251024 19:48:00.700656 1774 ts_tablet_manager.cc:1916] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.701146 2034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=13 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251024 19:48:00.701200 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_641c979b063f419884a09e448c68a1b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.701871 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_641c979b063f419884a09e448c68a1b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.702065 2034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Bootstrap complete.
I20251024 19:48:00.703596 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_66accd4024cc4cd98f10a25df857d785" } new_table_name: "default.Table_9ecacbf3e0fe41b7b54b20dbd81c4e05" modify_external_catalogs: true
I20251024 19:48:00.705191 1766 tablet_bootstrap.cc:492] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:00.705078 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3be1c2c5954947969e6903a3f016d589"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:00.705554 1766 ts_tablet_manager.cc:1403] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.003s
W20251024 19:48:00.705670 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3be1c2c5954947969e6903a3f016d589 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.706733 1766 raft_consensus.cc:359] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.707526 1766 raft_consensus.cc:385] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.707655 1766 raft_consensus.cc:740] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.707995 1766 consensus_queue.cc:260] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.708070 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_09c1a7fb5d6e47da800d84d81b233bce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.708626 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09c1a7fb5d6e47da800d84d81b233bce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.709421 1766 ts_tablet_manager.cc:1434] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251024 19:48:00.709784 1766 tablet_bootstrap.cc:492] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:00.712029 1766 tablet_bootstrap.cc:654] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.724335 2020 consensus_queue.cc:1048] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:00.725348 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 0397637f0778442a871528d5dd173689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.725696 2034 raft_consensus.cc:359] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:48:00.733228 1766 tablet_bootstrap.cc:492] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:00.733741 1766 ts_tablet_manager.cc:1403] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.006s
I20251024 19:48:00.735332 1766 raft_consensus.cc:359] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.735538 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 94fc8c5716af47faa4e831f05fce8610 (DEFAULT_TABLE table=default.Table_023e47eac6c24e328d5a01fcf3b2f579 [id=048d560cd32a44c5abf12bde27fe35c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.735816 1766 raft_consensus.cc:385] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.735992 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94fc8c5716af47faa4e831f05fce8610. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.736121 1766 raft_consensus.cc:740] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.736603 1766 consensus_queue.cc:260] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.737565 1766 ts_tablet_manager.cc:1434] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20251024 19:48:00.740195 1766 tablet_bootstrap.cc:492] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:00.742558 1766 tablet_bootstrap.cc:654] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.753084 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7966ff9879d24254aed39562b2f127d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.753981 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7966ff9879d24254aed39562b2f127d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.755870 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_cb99a44a90e849be948f4a481c07a91d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.756537 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb99a44a90e849be948f4a481c07a91d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.756980 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.757467 1766 tablet_bootstrap.cc:492] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:00.757723 1766 ts_tablet_manager.cc:1403] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.003s
I20251024 19:48:00.759152 2034 raft_consensus.cc:740] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8906492b1a4368936fac95c11e1f73, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.760277 2034 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:48:00.761026 1767 ts_tablet_manager.cc:1434] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.276s user 0.024s sys 0.040s
I20251024 19:48:00.761437 1767 tablet_bootstrap.cc:492] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:00.761811 1766 raft_consensus.cc:359] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.762382 1766 raft_consensus.cc:385] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.762544 1766 raft_consensus.cc:740] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.693150 1775 ts_tablet_manager.cc:1916] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.763499 1767 tablet_bootstrap.cc:654] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.766462 2059 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:48:00.769026 2059 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 19:48:00.768879 2034 sys_catalog.cc:565] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 19:48:00.771075 1767 tablet_bootstrap.cc:492] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:00.771252 1767 ts_tablet_manager.cc:1403] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20251024 19:48:00.762797 1766 consensus_queue.cc:260] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.766219 1486 tablet_service.cc:1505] Processing CreateTablet for tablet cb0405765c6146cbbb187af96a60be2e (DEFAULT_TABLE table=default.Table_3f7f7b5dfa674b19ad19c339c619326c [id=7faf1f110af34aa8939ad42e64f60011]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.773293 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb0405765c6146cbbb187af96a60be2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.775403 1617 tablet_service.cc:1505] Processing CreateTablet for tablet cb0405765c6146cbbb187af96a60be2e (DEFAULT_TABLE table=default.Table_3f7f7b5dfa674b19ad19c339c619326c [id=7faf1f110af34aa8939ad42e64f60011]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.775839 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb0405765c6146cbbb187af96a60be2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.771140 2020 raft_consensus.cc:493] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:00.776495 2020 raft_consensus.cc:515] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.777274 2020 leader_election.cc:290] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:00.778370 1447 leader_election.cc:343] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 6f21ffa44d7f48d08ed17ebf6c8ff29c
W20251024 19:48:00.781908 1444 leader_election.cc:343] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 6f21ffa44d7f48d08ed17ebf6c8ff29c
I20251024 19:48:00.782063 1444 leader_election.cc:304] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:00.782524 2020 raft_consensus.cc:2749] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:00.789319 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8d1e5bb0a09942ac81957d503c4b78b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.790212 2045 log.cc:927] Time spent T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97: Append to log took a long time: real 0.186s user 0.001s sys 0.000s
I20251024 19:48:00.793859 1766 ts_tablet_manager.cc:1434] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.033s user 0.002s sys 0.006s
I20251024 19:48:00.794351 1766 tablet_bootstrap.cc:492] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:00.797477 1766 tablet_bootstrap.cc:654] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.793185 1767 raft_consensus.cc:359] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.800897 1767 raft_consensus.cc:385] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.801028 1767 raft_consensus.cc:740] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.801422 1767 consensus_queue.cc:260] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.801891 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 230606a0ce084b7bab0e41e48cf8fa89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.802110 1767 ts_tablet_manager.cc:1434] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
I20251024 19:48:00.802255 1345 tablet_service.cc:1505] Processing CreateTablet for tablet cb0405765c6146cbbb187af96a60be2e (DEFAULT_TABLE table=default.Table_3f7f7b5dfa674b19ad19c339c619326c [id=7faf1f110af34aa8939ad42e64f60011]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:00.802466 1767 tablet_bootstrap.cc:492] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:00.803064 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb0405765c6146cbbb187af96a60be2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:00.805980 1766 tablet_bootstrap.cc:492] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:00.806272 1766 ts_tablet_manager.cc:1403] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20251024 19:48:00.807309 1766 raft_consensus.cc:359] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.808786 1766 raft_consensus.cc:385] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.809063 1766 raft_consensus.cc:740] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.809520 1766 consensus_queue.cc:260] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.811089 1766 ts_tablet_manager.cc:1434] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:48:00.811326 1767 tablet_bootstrap.cc:654] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.811985 1766 tablet_bootstrap.cc:492] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:00.809543 1768 tablet_bootstrap.cc:492] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:00.812999 1768 ts_tablet_manager.cc:1403] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.240s user 0.001s sys 0.017s
I20251024 19:48:00.814072 2026 raft_consensus.cc:493] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:00.814240 2026 raft_consensus.cc:515] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.814992 2026 leader_election.cc:290] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:00.815565 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c292ce055def4964ab143252c6b2b87e" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:00.816291 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c292ce055def4964ab143252c6b2b87e" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:00.816903 1305 leader_election.cc:343] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:00.813964 1768 raft_consensus.cc:359] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:00.817268 1308 leader_election.cc:343] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:00.817411 1308 leader_election.cc:304] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:00.817823 2026 raft_consensus.cc:2749] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:00.818060 1768 raft_consensus.cc:385] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.818186 1768 raft_consensus.cc:740] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.818493 1768 consensus_queue.cc:260] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.819383 1766 tablet_bootstrap.cc:654] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.820638 1768 ts_tablet_manager.cc:1434] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251024 19:48:00.821017 1768 tablet_bootstrap.cc:492] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:00.824173 1768 tablet_bootstrap.cc:654] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.825824 1767 tablet_bootstrap.cc:492] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:00.826030 1767 ts_tablet_manager.cc:1403] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20251024 19:48:00.827217 1767 raft_consensus.cc:359] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.827478 1767 raft_consensus.cc:385] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.827562 1767 raft_consensus.cc:740] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.827862 1767 consensus_queue.cc:260] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.831326 1767 ts_tablet_manager.cc:1434] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251024 19:48:00.838131 1767 tablet_bootstrap.cc:492] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:00.843153 1766 tablet_bootstrap.cc:492] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:00.843351 1766 ts_tablet_manager.cc:1403] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20251024 19:48:00.844372 1766 raft_consensus.cc:359] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.844601 1766 raft_consensus.cc:385] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.844710 1766 raft_consensus.cc:740] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.844998 1766 consensus_queue.cc:260] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.845727 1766 ts_tablet_manager.cc:1434] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251024 19:48:00.846104 1766 tablet_bootstrap.cc:492] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:00.848297 1766 tablet_bootstrap.cc:654] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.849802 1767 tablet_bootstrap.cc:654] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:00.836585 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e29abfa3bc8944aea7d1acd03e3eb897 consensus state has no leader
W20251024 19:48:00.837833 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d38154d3ffd34f4b8d7305b1c23a4fae consensus state has no leader
I20251024 19:48:00.858693 1557 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:00.858095 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f with delete_type TABLET_DATA_DELETED (replaced by tablet 9a1031d785b646b095ece9be666d2a77) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.869899 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]). Expected version 3 got 0
I20251024 19:48:00.870051 1767 tablet_bootstrap.cc:492] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:00.870236 1774 ts_tablet_manager.cc:1929] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:00.870251 1767 ts_tablet_manager.cc:1403] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.005s
I20251024 19:48:00.870445 1774 log.cc:1199] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/fb17a35d8d8446cf9870d05e107ed237
I20251024 19:48:00.870880 1774 ts_tablet_manager.cc:1950] T fb17a35d8d8446cf9870d05e107ed237 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:00.871372 1767 raft_consensus.cc:359] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:00.871613 1767 raft_consensus.cc:385] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.871722 1767 raft_consensus.cc:740] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.871960 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 133) took 3056 ms. Trace:
I20251024 19:48:00.872023 1767 consensus_queue.cc:260] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:00.872081 1774 rpcz_store.cc:276] 1024 19:47:57.815889 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:57.816055 (+ 166us) service_pool.cc:225] Handling call
1024 19:48:00.681226 (+2865171us) ts_tablet_manager.cc:1151] Deleting tablet fb17a35d8d8446cf9870d05e107ed237
1024 19:48:00.681326 (+ 100us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:00.811000 (+129674us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.868955 (+ 57955us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.871900 (+ 2945us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2854182}
I20251024 19:48:00.873432 1767 ts_tablet_manager.cc:1434] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:00.873852 1767 tablet_bootstrap.cc:492] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:00.875630 1696 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:00.876241 1767 tablet_bootstrap.cc:654] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.880589 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5f6d82174b9d4f64bbf61f72e5b58a25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.881448 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f6d82174b9d4f64bbf61f72e5b58a25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.886286 1774 tablet_replica.cc:333] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:00.886868 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9291fbfe8e484be094f28ad94814b13f" } new_table_name: "default.Table_aa7c283553a841e9ab2fa21fb6485a11" modify_external_catalogs: true
I20251024 19:48:00.889242 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 95d7fbfa8ae54da49dd5603294faa7d3 with delete_type TABLET_DATA_DELETED (replaced by tablet dc1b6f0a377945089c50012f74d38564) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.889748 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 8721ee81e4e24153a0aad8ce99f872e1 with delete_type TABLET_DATA_DELETED (replaced by tablet b79801594e95434cbb8fe7b75b14602e) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:00.892372 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d1e5bb0a09942ac81957d503c4b78b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.895195 2062 consensus_queue.cc:1048] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20251024 19:48:00.896029 2020 raft_consensus.cc:493] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:00.896220 2020 raft_consensus.cc:515] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.897548 2020 leader_election.cc:290] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:00.901857 1447 leader_election.cc:343] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 0397637f0778442a871528d5dd173689
W20251024 19:48:00.903040 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e03c488d0f264684b0897a81a04d7466 consensus state has no leader
I20251024 19:48:00.909775 1766 tablet_bootstrap.cc:492] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:00.910099 1766 ts_tablet_manager.cc:1403] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.020s
I20251024 19:48:00.911567 1766 raft_consensus.cc:359] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.912434 1766 raft_consensus.cc:385] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.912671 1766 raft_consensus.cc:740] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.913166 1766 consensus_queue.cc:260] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.889762 1774 raft_consensus.cc:2243] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:00.915006 1774 raft_consensus.cc:2272] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:00.914561 1766 ts_tablet_manager.cc:1434] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251024 19:48:00.915714 1774 ts_tablet_manager.cc:1916] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.916039 1766 tablet_bootstrap.cc:492] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:00.919396 1766 tablet_bootstrap.cc:654] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.922458 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 3caecdff61ce48f7b7c3846bb380d21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:00.924260 1774 ts_tablet_manager.cc:1929] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:00.924403 1774 log.cc:1199] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/2e956ea914c545d79be6d632bfc4ec57
I20251024 19:48:00.924832 1774 ts_tablet_manager.cc:1950] T 2e956ea914c545d79be6d632bfc4ec57 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:00.925170 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 143) took 2872 ms. Trace:
I20251024 19:48:00.925284 1774 rpcz_store.cc:276] 1024 19:47:58.052604 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.052941 (+ 337us) service_pool.cc:225] Handling call
1024 19:48:00.886182 (+2833241us) ts_tablet_manager.cc:1151] Deleting tablet 2e956ea914c545d79be6d632bfc4ec57
1024 19:48:00.886196 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:00.920120 (+ 33924us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.924121 (+ 4001us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:00.925131 (+ 1010us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2818143}
I20251024 19:48:00.889137 1768 tablet_bootstrap.cc:492] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:00.927577 1768 ts_tablet_manager.cc:1403] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.002s
I20251024 19:48:00.928378 1768 raft_consensus.cc:359] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.928553 1768 raft_consensus.cc:385] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.928642 1768 raft_consensus.cc:740] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.930425 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a5cf2d40da674fc592fd529d58dc0575"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:00.931097 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 2e956ea914c545d79be6d632bfc4ec57 (table default.Table_3819751f36bf419e9edd0f1ff339713a [id=350b37bf42194fcfbb660bffe83e47c4]) successfully deleted
I20251024 19:48:00.931205 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2952ae2bfd4d4924afd95ddf2e8404bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:00.931944 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2952ae2bfd4d4924afd95ddf2e8404bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:00.935276 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5cf2d40da674fc592fd529d58dc0575 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:00.941519 1766 tablet_bootstrap.cc:492] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:00.942831 1766 ts_tablet_manager.cc:1403] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.006s
I20251024 19:48:00.944170 1766 raft_consensus.cc:359] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.946314 1766 raft_consensus.cc:385] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.947144 1766 raft_consensus.cc:740] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.948531 1766 consensus_queue.cc:260] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.950701 1766 ts_tablet_manager.cc:1434] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251024 19:48:00.951367 1766 tablet_bootstrap.cc:492] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:00.966570 1776 tablet_replica.cc:333] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:48:00.966795 1444 leader_election.cc:343] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 0397637f0778442a871528d5dd173689
I20251024 19:48:00.966956 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet cc0bf7aee1a1421b9d985020a1674639 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:00.966979 1444 leader_election.cc:304] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:00.967634 2020 raft_consensus.cc:2749] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:00.969148 1776 raft_consensus.cc:2243] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:00.969367 1776 raft_consensus.cc:2272] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:00.970449 1766 tablet_bootstrap.cc:654] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:00.970525 1776 ts_tablet_manager.cc:1916] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:00.979522 1766 tablet_bootstrap.cc:492] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:00.979775 1766 ts_tablet_manager.cc:1403] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.001s
I20251024 19:48:00.983107 1766 raft_consensus.cc:359] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.984534 1766 raft_consensus.cc:385] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:00.984884 1766 raft_consensus.cc:740] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:00.985618 1766 consensus_queue.cc:260] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:00.987872 1766 ts_tablet_manager.cc:1434] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20251024 19:48:00.988878 1766 tablet_bootstrap.cc:492] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:00.992262 1766 tablet_bootstrap.cc:654] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.001531 1776 ts_tablet_manager.cc:1929] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.001904 1776 log.cc:1199] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/95d7fbfa8ae54da49dd5603294faa7d3
I20251024 19:48:01.002672 1776 ts_tablet_manager.cc:1950] T 95d7fbfa8ae54da49dd5603294faa7d3 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:01.018548 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 144) took 2915 ms. Trace:
I20251024 19:48:01.018708 1774 rpcz_store.cc:276] 1024 19:47:58.121045 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.121151 (+ 106us) service_pool.cc:225] Handling call
1024 19:48:01.018306 (+2897155us) ts_tablet_manager.cc:1151] Deleting tablet f346a90a9efe442f82750336e3893deb
1024 19:48:01.018323 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:01.018484 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2796285}
I20251024 19:48:01.019090 1774 tablet_replica.cc:333] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:01.019321 1774 raft_consensus.cc:2243] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.019529 1774 raft_consensus.cc:2272] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.020200 1774 ts_tablet_manager.cc:1916] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.022233 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 95d7fbfa8ae54da49dd5603294faa7d3 (table default.Table_52ec55c23c394294a68850ab6b33aabe [id=fb0e7cab20f44aaebdb09862bd193f41]) successfully deleted
W20251024 19:48:01.022513 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet f346a90a9efe442f82750336e3893deb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f346a90a9efe442f82750336e3893deb already in progress: creating tablet
I20251024 19:48:00.992107 2026 raft_consensus.cc:493] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.022888 2026 raft_consensus.cc:515] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.023941 2026 leader_election.cc:290] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:00.956609 1768 consensus_queue.cc:260] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.025132 1768 ts_tablet_manager.cc:1434] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.097s user 0.001s sys 0.001s
I20251024 19:48:01.025466 1768 tablet_bootstrap.cc:492] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:01.028301 1768 tablet_bootstrap.cc:654] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:01.031056 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2496f103436047dea5dabb577651249f consensus state has no leader
I20251024 19:48:01.033131 1187 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]). Expected version 3 got 0
I20251024 19:48:01.035281 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet 16bc737104ab46b3b071a0760dbc847d from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:01.041010 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet a48249f1824a40b087b80a563816d485 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:01.041864 1776 tablet_replica.cc:333] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:01.042066 1189 master_service.cc:954] ReplaceTablet: received request to replace tablet 369cebfda6ee43638b79f8652fa55855 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:01.042785 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dc366a1f70d84d4f8276c1872b9fa0f7" } modify_external_catalogs: true
I20251024 19:48:01.042996 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dc366a1f70d84d4f8276c1872b9fa0f7" } modify_external_catalogs: true
I20251024 19:48:01.043821 1768 tablet_bootstrap.cc:492] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:01.044028 1768 ts_tablet_manager.cc:1403] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20251024 19:48:01.044605 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6eb10ae51f3a43c2ab15f362708fba94" } modify_external_catalogs: true
I20251024 19:48:01.044780 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6eb10ae51f3a43c2ab15f362708fba94" } modify_external_catalogs: true
I20251024 19:48:01.046018 1768 raft_consensus.cc:359] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.046278 1768 raft_consensus.cc:385] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.046362 1768 raft_consensus.cc:740] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.046677 1768 consensus_queue.cc:260] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.047302 1768 ts_tablet_manager.cc:1434] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251024 19:48:01.051911 1768 tablet_bootstrap.cc:492] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:01.054064 1768 tablet_bootstrap.cc:654] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:01.064766 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b79801594e95434cbb8fe7b75b14602e consensus state has no leader
I20251024 19:48:01.083563 1774 ts_tablet_manager.cc:1929] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.083762 1774 log.cc:1199] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/b2ea545b2cc14f589eda9419e55234bf
I20251024 19:48:01.084244 1774 ts_tablet_manager.cc:1950] T b2ea545b2cc14f589eda9419e55234bf P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:01.087570 1775 ts_tablet_manager.cc:1929] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.087877 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe97e31310ae4733ba93de1b62a37842" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:01.088568 1308 leader_election.cc:343] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:01.092566 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet cc0bf7aee1a1421b9d985020a1674639 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.092761 2020 raft_consensus.cc:493] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.092941 2020 raft_consensus.cc:515] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.094985 2020 leader_election.cc:290] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:01.095198 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 145) took 2965 ms. Trace:
I20251024 19:48:01.095314 1774 rpcz_store.cc:276] 1024 19:47:58.129414 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.129617 (+ 203us) service_pool.cc:225] Handling call
1024 19:48:01.019012 (+2889395us) ts_tablet_manager.cc:1151] Deleting tablet b2ea545b2cc14f589eda9419e55234bf
1024 19:48:01.019021 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:01.071514 (+ 52493us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:01.083417 (+ 11903us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:01.095128 (+ 11711us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2886291}
I20251024 19:48:01.095657 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 8721ee81e4e24153a0aad8ce99f872e1 with delete_type TABLET_DATA_DELETED (replaced by tablet b79801594e95434cbb8fe7b75b14602e) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.095805 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet b2ea545b2cc14f589eda9419e55234bf (table default.Table_74349eed6d4b4e8e95ae01909bbb91e9 [id=94b8daec91b6478f9dc43734542246e0]) successfully deleted
I20251024 19:48:01.095332 1775 log.cc:1199] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/65f646bd05274cb6859c0b7f92b61352
W20251024 19:48:01.096648 1447 leader_election.cc:343] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 95d7fbfa8ae54da49dd5603294faa7d3
I20251024 19:48:01.101112 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 146) took 2927 ms. Trace:
I20251024 19:48:01.101233 1774 rpcz_store.cc:276] 1024 19:47:58.173558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.173655 (+ 97us) service_pool.cc:225] Handling call
1024 19:48:01.100862 (+2927207us) ts_tablet_manager.cc:1151] Deleting tablet b5473699c1b647f8b86ee43fdb56bdfd
1024 19:48:01.100877 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:01.101035 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2922502}
W20251024 19:48:01.101744 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b5473699c1b647f8b86ee43fdb56bdfd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5473699c1b647f8b86ee43fdb56bdfd already in progress: creating tablet
I20251024 19:48:01.102077 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e956ea914c545d79be6d632bfc4ec57 with delete_type TABLET_DATA_DELETED (replaced by tablet a3259823b622461fa823841925b3d203) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.102459 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe97e31310ae4733ba93de1b62a37842" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:01.103184 1305 leader_election.cc:343] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:01.103359 1305 leader_election.cc:304] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:01.104130 2026 raft_consensus.cc:2749] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:01.104332 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 172) took 2374 ms. Trace:
I20251024 19:48:01.104410 1774 rpcz_store.cc:276] 1024 19:47:58.729895 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.730230 (+ 335us) service_pool.cc:225] Handling call
1024 19:48:01.104175 (+2373945us) ts_tablet_manager.cc:1151] Deleting tablet 897db285f5c84930a5b438be1cb4eb96
1024 19:48:01.104186 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:01.104277 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2362069}
I20251024 19:48:01.104719 1774 tablet_replica.cc:333] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:01.105012 1774 raft_consensus.cc:2243] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.105350 1774 raft_consensus.cc:2272] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:01.106931 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 897db285f5c84930a5b438be1cb4eb96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 897db285f5c84930a5b438be1cb4eb96 already in progress: creating tablet
I20251024 19:48:01.107625 1776 raft_consensus.cc:2243] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.107798 1774 ts_tablet_manager.cc:1916] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.108067 1776 raft_consensus.cc:2272] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.117648 1776 ts_tablet_manager.cc:1916] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.123788 1767 tablet_bootstrap.cc:492] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:01.123772 1774 ts_tablet_manager.cc:1929] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.123929 1774 log.cc:1199] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/90405a0c13bd4c229b011c90232f751d
I20251024 19:48:01.124076 1767 ts_tablet_manager.cc:1403] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.250s user 0.059s sys 0.075s
I20251024 19:48:01.124372 1774 ts_tablet_manager.cc:1950] T 90405a0c13bd4c229b011c90232f751d P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:01.125224 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 173) took 2395 ms. Trace:
I20251024 19:48:01.125350 1774 rpcz_store.cc:276] 1024 19:47:58.730173 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.730442 (+ 269us) service_pool.cc:225] Handling call
1024 19:48:01.104637 (+2374195us) ts_tablet_manager.cc:1151] Deleting tablet 90405a0c13bd4c229b011c90232f751d
1024 19:48:01.104644 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:01.116777 (+ 12133us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:01.123589 (+ 6812us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:01.125171 (+ 1582us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2362412}
I20251024 19:48:01.125418 1767 raft_consensus.cc:359] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.125794 1767 raft_consensus.cc:385] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.126132 1767 raft_consensus.cc:740] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.126633 1767 consensus_queue.cc:260] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.127537 1767 ts_tablet_manager.cc:1434] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20251024 19:48:01.127987 1767 tablet_bootstrap.cc:492] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:01.130300 1767 tablet_bootstrap.cc:654] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.130529 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 90405a0c13bd4c229b011c90232f751d (table default.Table_eb9c8019451d46fba31a87a0b6ac6ad8 [id=e5b1dd31c2114e8a95ba33981029255c]) successfully deleted
I20251024 19:48:01.131263 1774 tablet_replica.cc:333] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:01.131573 1774 raft_consensus.cc:2243] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.131778 1774 raft_consensus.cc:2272] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.132550 1774 ts_tablet_manager.cc:1916] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.137126 1767 tablet_bootstrap.cc:492] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:01.137271 1767 ts_tablet_manager.cc:1403] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20251024 19:48:01.138181 1767 raft_consensus.cc:359] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.138357 1767 raft_consensus.cc:385] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.138449 1767 raft_consensus.cc:740] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.138707 1767 consensus_queue.cc:260] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.138912 2020 raft_consensus.cc:493] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.139493 1767 ts_tablet_manager.cc:1434] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:01.139793 1767 tablet_bootstrap.cc:492] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:01.141899 1767 tablet_bootstrap.cc:654] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:01.144692 1444 leader_election.cc:343] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 95d7fbfa8ae54da49dd5603294faa7d3
I20251024 19:48:01.145085 1444 leader_election.cc:304] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:01.144696 2020 raft_consensus.cc:515] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.149123 2062 raft_consensus.cc:2749] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:01.149340 1444 leader_election.cc:343] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 110907ae943f45b5bcaa9b66ccdb84fd
I20251024 19:48:01.149533 2020 leader_election.cc:290] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:01.149852 1767 tablet_bootstrap.cc:492] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:01.150029 1767 ts_tablet_manager.cc:1403] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20251024 19:48:01.149652 1775 ts_tablet_manager.cc:1950] T 65f646bd05274cb6859c0b7f92b61352 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:01.150869 1767 raft_consensus.cc:359] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.151046 1767 raft_consensus.cc:385] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.151051 1775 tablet_replica.cc:333] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:01.151105 1767 raft_consensus.cc:740] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.151257 1775 raft_consensus.cc:2243] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.151432 1775 raft_consensus.cc:2272] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.151316 1767 consensus_queue.cc:260] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.149871 1766 tablet_bootstrap.cc:492] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:01.151747 1766 ts_tablet_manager.cc:1403] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.163s user 0.024s sys 0.032s
W20251024 19:48:01.150023 1447 leader_election.cc:343] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 110907ae943f45b5bcaa9b66ccdb84fd
I20251024 19:48:01.151960 1447 leader_election.cc:304] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:01.152019 1767 ts_tablet_manager.cc:1434] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251024 19:48:01.152447 1767 tablet_bootstrap.cc:492] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:01.152529 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 65f646bd05274cb6859c0b7f92b61352 (table default.Table_d09f149249c44c50adcb71321c133f2d [id=665389c7bde54b7786edabcb91a9b33b]) successfully deleted
I20251024 19:48:01.153050 2020 raft_consensus.cc:2749] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:01.153424 1775 ts_tablet_manager.cc:1916] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.153954 1766 raft_consensus.cc:359] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:01.154138 1766 raft_consensus.cc:385] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.154222 1766 raft_consensus.cc:740] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.154422 1766 consensus_queue.cc:260] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:01.154747 1767 tablet_bootstrap.cc:654] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.154902 1766 ts_tablet_manager.cc:1434] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:01.155275 1766 tablet_bootstrap.cc:492] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:01.157109 1766 tablet_bootstrap.cc:654] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.158502 1774 ts_tablet_manager.cc:1929] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.159355 1774 log.cc:1199] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/8721ee81e4e24153a0aad8ce99f872e1
I20251024 19:48:01.160027 1774 ts_tablet_manager.cc:1950] T 8721ee81e4e24153a0aad8ce99f872e1 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:01.161707 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 174) took 2430 ms. Trace:
I20251024 19:48:01.163051 1774 rpcz_store.cc:276] 1024 19:47:58.731104 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.731187 (+ 83us) service_pool.cc:225] Handling call
1024 19:48:01.131179 (+2399992us) ts_tablet_manager.cc:1151] Deleting tablet 8721ee81e4e24153a0aad8ce99f872e1
1024 19:48:01.131189 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:01.142563 (+ 11374us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:01.158363 (+ 15800us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:01.161617 (+ 3254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2388719}
I20251024 19:48:01.164131 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 8721ee81e4e24153a0aad8ce99f872e1 (table default.Table_aa7c283553a841e9ab2fa21fb6485a11 [id=7ec9391c52b1424eb52850f2c6ba4779]) successfully deleted
I20251024 19:48:01.164984 1774 tablet_replica.cc:333] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:01.165019 1511 raft_consensus.cc:1275] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Refusing update from remote peer af3952de050140d383f6b0de7666fa97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 19:48:01.165594 1887 consensus_queue.cc:1048] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:01.166185 1774 raft_consensus.cc:2243] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.167129 1372 raft_consensus.cc:1275] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Refusing update from remote peer af3952de050140d383f6b0de7666fa97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 19:48:01.168489 1887 consensus_queue.cc:1048] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:01.168998 1774 raft_consensus.cc:2272] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.170850 2026 raft_consensus.cc:493] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.171967 2026 raft_consensus.cc:515] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.172891 2026 leader_election.cc:290] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:01.174387 2020 raft_consensus.cc:493] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.174619 2020 raft_consensus.cc:515] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.174702 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "979dd87a8c66480ea314fcc75fa878c3" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:01.175037 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "979dd87a8c66480ea314fcc75fa878c3" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:01.175246 2020 leader_election.cc:290] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:01.175341 1648 raft_consensus.cc:2468] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:01.175705 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc0bf7aee1a1421b9d985020a1674639" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:01.175935 1650 raft_consensus.cc:2468] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
W20251024 19:48:01.176985 1447 leader_election.cc:343] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: cc0bf7aee1a1421b9d985020a1674639
I20251024 19:48:01.178450 1444 leader_election.cc:304] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:01.178910 2020 raft_consensus.cc:2804] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251024 19:48:01.178947 1308 leader_election.cc:343] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:01.179047 2020 raft_consensus.cc:493] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:01.179255 2020 raft_consensus.cc:3060] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:01.180097 2082 mvcc.cc:204] Tried to move back new op lower bound from 7214429311644401664 to 7214429309055578112. Current Snapshot: MvccSnapshot[applied={T|T < 7214429311644401664}]
I20251024 19:48:01.181341 1305 leader_election.cc:304] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
I20251024 19:48:01.182133 2026 raft_consensus.cc:2804] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:01.182271 2026 raft_consensus.cc:493] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:01.182212 2020 raft_consensus.cc:515] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.182404 2026 raft_consensus.cc:3060] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:01.183790 1447 leader_election.cc:343] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: cc0bf7aee1a1421b9d985020a1674639
I20251024 19:48:01.183837 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc0bf7aee1a1421b9d985020a1674639" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:01.184055 1650 raft_consensus.cc:3060] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:01.189008 2020 leader_election.cc:290] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:01.189021 1650 raft_consensus.cc:2468] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:01.189318 2026 raft_consensus.cc:515] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.189484 1444 leader_election.cc:304] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:01.190301 2020 raft_consensus.cc:2804] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:01.190541 2020 raft_consensus.cc:697] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:01.190979 2020 consensus_queue.cc:237] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.191435 1766 tablet_bootstrap.cc:492] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:01.191462 1775 ts_tablet_manager.cc:1929] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.191620 1775 log.cc:1199] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/0397637f0778442a871528d5dd173689
I20251024 19:48:01.191906 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "979dd87a8c66480ea314fcc75fa878c3" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:01.192276 1650 raft_consensus.cc:3060] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:01.191637 1766 ts_tablet_manager.cc:1403] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.001s
I20251024 19:48:01.192992 1775 ts_tablet_manager.cc:1950] T 0397637f0778442a871528d5dd173689 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:01.193611 1775 tablet_replica.cc:333] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:01.193755 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 0397637f0778442a871528d5dd173689 (table default.Table_59f512ddad6842eca4865d6387dee631 [id=e6d5a0f628404ad0bd19cc019911e269]) successfully deleted
I20251024 19:48:01.193859 1775 raft_consensus.cc:2243] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.194159 1775 raft_consensus.cc:2272] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.194156 1766 raft_consensus.cc:359] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.194394 1766 raft_consensus.cc:385] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.194473 1766 raft_consensus.cc:740] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.194536 2100 tablet.cc:1722] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:01.194824 1766 consensus_queue.cc:260] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.195621 2026 leader_election.cc:290] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:01.195880 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "979dd87a8c66480ea314fcc75fa878c3" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
W20251024 19:48:01.196400 1308 leader_election.cc:343] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:01.196887 1766 ts_tablet_manager.cc:1434] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251024 19:48:01.197170 1766 tablet_bootstrap.cc:492] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:01.200860 1650 raft_consensus.cc:2468] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:01.201592 1305 leader_election.cc:304] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
I20251024 19:48:01.204111 1775 ts_tablet_manager.cc:1916] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.202226 2026 raft_consensus.cc:2804] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:01.207978 2026 raft_consensus.cc:697] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:01.208669 2026 consensus_queue.cc:237] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.210952 2101 tablet.cc:1722] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:01.212652 2048 raft_consensus.cc:493] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.212913 2048 raft_consensus.cc:515] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.213855 2048 leader_election.cc:290] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:01.217020 1586 leader_election.cc:343] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: b2ea545b2cc14f589eda9419e55234bf
I20251024 19:48:01.218037 1767 tablet_bootstrap.cc:492] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:01.218201 1767 ts_tablet_manager.cc:1403] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.033s
I20251024 19:48:01.219071 1767 raft_consensus.cc:359] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.219277 1767 raft_consensus.cc:385] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.219390 1767 raft_consensus.cc:740] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.214501 1774 ts_tablet_manager.cc:1916] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.219756 1767 consensus_queue.cc:260] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.220261 1767 ts_tablet_manager.cc:1434] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:01.220598 1767 tablet_bootstrap.cc:492] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:01.222784 1767 tablet_bootstrap.cc:654] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.244727 1183 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]). Expected version 3 got 0
I20251024 19:48:01.245440 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet 639836c656f44a819036f4d00e055bd9 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:01.245956 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_31fb0d4536b94892aeceb91307c74837" } modify_external_catalogs: true
I20251024 19:48:01.246147 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_31fb0d4536b94892aeceb91307c74837" } modify_external_catalogs: true
I20251024 19:48:01.247377 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_eb9c8019451d46fba31a87a0b6ac6ad8" } modify_external_catalogs: true
I20251024 19:48:01.247691 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_eb9c8019451d46fba31a87a0b6ac6ad8" } modify_external_catalogs: true
I20251024 19:48:01.254755 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]). Expected version 3 got 0
I20251024 19:48:01.255950 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d696d859aaad48ceac033959d75bcf11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.256778 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d696d859aaad48ceac033959d75bcf11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.266728 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1c89d244042a4852a9cb4113f6726332" } modify_external_catalogs: true
I20251024 19:48:01.267341 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1c89d244042a4852a9cb4113f6726332" } modify_external_catalogs: true
I20251024 19:48:01.300665 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]). Expected version 3 got 0
I20251024 19:48:01.302512 1774 ts_tablet_manager.cc:1929] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.302548 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 6424b2347ef642feb38a9596f2d2be1d (DEFAULT_TABLE table=default.Table_da1c2655e0de479e91cf0a21e1a728b5 [id=42ffad851c714cb28e672b448efb33bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.302740 1774 log.cc:1199] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/f91316c030dc4f7da27977a9dde01f4b
I20251024 19:48:01.303066 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6424b2347ef642feb38a9596f2d2be1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.305225 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 6424b2347ef642feb38a9596f2d2be1d (DEFAULT_TABLE table=default.Table_da1c2655e0de479e91cf0a21e1a728b5 [id=42ffad851c714cb28e672b448efb33bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.305799 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6424b2347ef642feb38a9596f2d2be1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.308954 1774 ts_tablet_manager.cc:1950] T f91316c030dc4f7da27977a9dde01f4b P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:01.310072 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 175) took 2514 ms. Trace:
I20251024 19:48:01.310212 1774 rpcz_store.cc:276] 1024 19:47:58.794623 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.794703 (+ 80us) service_pool.cc:225] Handling call
1024 19:48:01.164761 (+2370058us) ts_tablet_manager.cc:1151] Deleting tablet f91316c030dc4f7da27977a9dde01f4b
1024 19:48:01.164778 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:01.223524 (+ 58746us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:01.281238 (+ 57714us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:01.309319 (+ 28081us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2369653}
I20251024 19:48:01.310719 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet f91316c030dc4f7da27977a9dde01f4b (table default.Table_cdf2e8872b724c7aa86d5091eb52fb1c [id=a402387542654ccf8e0b04ca8ecebb1a]) successfully deleted
I20251024 19:48:01.311415 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf4596b333244cc9e9e35a128721e4a (DEFAULT_TABLE table=default.Table_25adc9559f2b43e2a4adc92dfc0c7807 [id=60d23d1c07704c33bc979f7ba025b102]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.311964 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf4596b333244cc9e9e35a128721e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.313462 2026 raft_consensus.cc:493] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.313628 2026 raft_consensus.cc:515] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.316239 2026 leader_election.cc:290] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:01.317226 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "346383e85cc64b3a8da70a71868ceccc" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:01.317904 1305 leader_election.cc:343] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:01.318562 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf4596b333244cc9e9e35a128721e4a (DEFAULT_TABLE table=default.Table_25adc9559f2b43e2a4adc92dfc0c7807 [id=60d23d1c07704c33bc979f7ba025b102]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.318866 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 81a3de69a13c459b9b7c02761ab7261e (DEFAULT_TABLE table=default.Table_bbd727eca62c408ca76741ebea86eb3b [id=3d82ad2f9e184043aed4ec0b05df724f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.318980 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 176) took 2512 ms. Trace:
I20251024 19:48:01.319236 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81a3de69a13c459b9b7c02761ab7261e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.319355 1774 rpcz_store.cc:276] 1024 19:47:58.799455 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.799517 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:01.311126 (+2511609us) ts_tablet_manager.cc:1151] Deleting tablet 5368789df6024d7daa9ccc2257137d58
1024 19:48:01.311140 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:01.311453 (+ 313us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2511107}
W20251024 19:48:01.320292 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 5368789df6024d7daa9ccc2257137d58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5368789df6024d7daa9ccc2257137d58 already in progress: creating tablet
I20251024 19:48:01.320976 1774 tablet_replica.cc:333] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:01.322063 1774 raft_consensus.cc:2243] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:01.323369 1774 pending_rounds.cc:70] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:01.323738 1774 pending_rounds.cc:77] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429301296807936 op_type: NO_OP noop_request { }
I20251024 19:48:01.324280 1774 raft_consensus.cc:2889] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:01.325416 1774 raft_consensus.cc:2272] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.327358 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 81a3de69a13c459b9b7c02761ab7261e (DEFAULT_TABLE table=default.Table_bbd727eca62c408ca76741ebea86eb3b [id=3d82ad2f9e184043aed4ec0b05df724f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.328063 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 129debb55f4a4b7295a8ad88e55ebc83 (DEFAULT_TABLE table=default.Table_c777799df90e491992909e8a45205a06 [id=d11ca7b446a44dd28c9c243a6d7787e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.328233 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf4596b333244cc9e9e35a128721e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.328419 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 129debb55f4a4b7295a8ad88e55ebc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.328900 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 129debb55f4a4b7295a8ad88e55ebc83 (DEFAULT_TABLE table=default.Table_c777799df90e491992909e8a45205a06 [id=d11ca7b446a44dd28c9c243a6d7787e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.329263 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 129debb55f4a4b7295a8ad88e55ebc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.330224 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 5b96fc2309274048bed7a35e4bf4bae2
I20251024 19:48:01.330899 1189 master_service.cc:954] ReplaceTablet: received request to replace tablet 047436af5f6d4f688bd83e33089d3ee4 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:01.333452 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81a3de69a13c459b9b7c02761ab7261e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.333909 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 16bc737104ab46b3b071a0760dbc847d
I20251024 19:48:01.334524 1184 catalog_manager.cc:6719] ReplaceTablet: tablet 16bc737104ab46b3b071a0760dbc847d deleted and replaced by tablet 385e71e672154e399f14321eac0d8625
I20251024 19:48:01.334973 1186 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_9764ce0360fd4fcfb03ab7df33ed112a [id=0fa321e2d4d847e69435f6de429d8779] alter complete (version 1)
I20251024 19:48:01.335649 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b96fc2309274048bed7a35e4bf4bae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.335915 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 16bc737104ab46b3b071a0760dbc847d with delete_type TABLET_DATA_DELETED (replaced by tablet 385e71e672154e399f14321eac0d8625) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.337433 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b96fc2309274048bed7a35e4bf4bae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:01.337752 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 16bc737104ab46b3b071a0760dbc847d with delete_type TABLET_DATA_DELETED (replaced by tablet 385e71e672154e399f14321eac0d8625) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:01.338536 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 979dd87a8c66480ea314fcc75fa878c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.338567 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ea545b2cc14f589eda9419e55234bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.338802 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet f91316c030dc4f7da27977a9dde01f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.339032 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2ea545b2cc14f589eda9419e55234bf" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:01.339530 1509 raft_consensus.cc:2468] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:01.340257 1586 leader_election.cc:304] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:01.340705 2048 raft_consensus.cc:2804] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:01.340826 2048 raft_consensus.cc:493] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:01.340920 2048 raft_consensus.cc:3060] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:01.341892 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 1c92b080805845c6af8f1cb160110915 (DEFAULT_TABLE table=default.Table_7b4b72b748a04135803b67edd2b45447 [id=4e0d6325c2ab4ea58d754bb01fd7cc85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.342370 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c92b080805845c6af8f1cb160110915. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.344079 2048 raft_consensus.cc:515] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.345716 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 1c92b080805845c6af8f1cb160110915 (DEFAULT_TABLE table=default.Table_7b4b72b748a04135803b67edd2b45447 [id=4e0d6325c2ab4ea58d754bb01fd7cc85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.348013 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c92b080805845c6af8f1cb160110915. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.349332 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 9a1031d785b646b095ece9be666d2a77 (DEFAULT_TABLE table=default.Table_a1d09325af844d71b290d95b8a4aaf12 [id=b62ec7bcad7c41b1b946e9ebf2f4bf39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.349705 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 2d01572e51694d9bb58c3a59d4205102 (DEFAULT_TABLE table=default.Table_740023c3917c4850b564d096c64670d4 [id=2ea13e78137d47429d7cbe374d2113ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.350399 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a1031d785b646b095ece9be666d2a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.350479 2048 leader_election.cc:290] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:01.350812 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d01572e51694d9bb58c3a59d4205102. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.351325 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "346383e85cc64b3a8da70a71868ceccc" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:01.351735 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 15c08dcf379f4df9a0e461d6dcfeeac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.357673 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 95d7fbfa8ae54da49dd5603294faa7d3 with delete_type TABLET_DATA_DELETED (replaced by tablet dc1b6f0a377945089c50012f74d38564) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:01.359160 1308 leader_election.cc:343] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:01.359373 1308 leader_election.cc:304] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:01.359678 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 110907ae943f45b5bcaa9b66ccdb84fd with delete_type TABLET_DATA_DELETED (replaced by tablet 1d04a9b54bd6446d8deb64292cef672d) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:01.360131 1586 leader_election.cc:343] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: b2ea545b2cc14f589eda9419e55234bf
I20251024 19:48:01.360471 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 50cea450eda9485daba40bf875a1b90c (DEFAULT_TABLE table=default.Table_3bd0db5c2861405a9dd9d66731cd9a57 [id=bfd41da632b94dcf90b0e0d230dc8fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.360903 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50cea450eda9485daba40bf875a1b90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.361992 1766 tablet_bootstrap.cc:654] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.355754 1776 ts_tablet_manager.cc:1929] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.362911 1776 log.cc:1199] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/8721ee81e4e24153a0aad8ce99f872e1
I20251024 19:48:01.363691 1776 ts_tablet_manager.cc:1950] T 8721ee81e4e24153a0aad8ce99f872e1 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:01.364228 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 9a1031d785b646b095ece9be666d2a77 (DEFAULT_TABLE table=default.Table_a1d09325af844d71b290d95b8a4aaf12 [id=b62ec7bcad7c41b1b946e9ebf2f4bf39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.364540 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 2d01572e51694d9bb58c3a59d4205102 (DEFAULT_TABLE table=default.Table_740023c3917c4850b564d096c64670d4 [id=2ea13e78137d47429d7cbe374d2113ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.366034 1774 ts_tablet_manager.cc:1916] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.366155 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 8721ee81e4e24153a0aad8ce99f872e1 (table default.Table_aa7c283553a841e9ab2fa21fb6485a11 [id=7ec9391c52b1424eb52850f2c6ba4779]) successfully deleted
I20251024 19:48:01.366902 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a1031d785b646b095ece9be666d2a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.367262 1775 ts_tablet_manager.cc:1929] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.367460 1775 log.cc:1199] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/6f21ffa44d7f48d08ed17ebf6c8ff29c
I20251024 19:48:01.367919 1775 ts_tablet_manager.cc:1950] T 6f21ffa44d7f48d08ed17ebf6c8ff29c P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:01.372048 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 81a3de69a13c459b9b7c02761ab7261e (DEFAULT_TABLE table=default.Table_bbd727eca62c408ca76741ebea86eb3b [id=3d82ad2f9e184043aed4ec0b05df724f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.372507 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 129debb55f4a4b7295a8ad88e55ebc83 (DEFAULT_TABLE table=default.Table_c777799df90e491992909e8a45205a06 [id=d11ca7b446a44dd28c9c243a6d7787e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.372608 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 9820e68ad5264df1a79b22f3b5369c5a with delete_type TABLET_DATA_DELETED (replaced by tablet e5f58c190a9a4c5391f1330f79865589) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.372750 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b96fc2309274048bed7a35e4bf4bae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.372993 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 16bc737104ab46b3b071a0760dbc847d with delete_type TABLET_DATA_DELETED (replaced by tablet 385e71e672154e399f14321eac0d8625) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.373361 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 1c92b080805845c6af8f1cb160110915 (DEFAULT_TABLE table=default.Table_7b4b72b748a04135803b67edd2b45447 [id=4e0d6325c2ab4ea58d754bb01fd7cc85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.374141 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 2d01572e51694d9bb58c3a59d4205102 (DEFAULT_TABLE table=default.Table_740023c3917c4850b564d096c64670d4 [id=2ea13e78137d47429d7cbe374d2113ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.374563 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf4596b333244cc9e9e35a128721e4a (DEFAULT_TABLE table=default.Table_25adc9559f2b43e2a4adc92dfc0c7807 [id=60d23d1c07704c33bc979f7ba025b102]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.375072 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 6424b2347ef642feb38a9596f2d2be1d (DEFAULT_TABLE table=default.Table_da1c2655e0de479e91cf0a21e1a728b5 [id=42ffad851c714cb28e672b448efb33bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.375491 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81a3de69a13c459b9b7c02761ab7261e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.375983 1624 tablet_service.cc:1505] Processing CreateTablet for tablet ee68243b4a664575b679950603bb250e (DEFAULT_TABLE table=default.Table_605117e2eb734e6b8dc34ceb4b5b5b71 [id=38b6e1273e1a43bd8b12604db5371677]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.376554 1776 tablet_replica.cc:333] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:01.376804 2073 mvcc.cc:204] Tried to move back new op lower bound from 7214429311644401664 to 7214429309055578112. Current Snapshot: MvccSnapshot[applied={T|T < 7214429311644401664}]
I20251024 19:48:01.377503 2062 raft_consensus.cc:493] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.377682 2062 raft_consensus.cc:515] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.377929 1484 tablet_service.cc:1505] Processing CreateTablet for tablet ee68243b4a664575b679950603bb250e (DEFAULT_TABLE table=default.Table_605117e2eb734e6b8dc34ceb4b5b5b71 [id=38b6e1273e1a43bd8b12604db5371677]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.378355 2062 leader_election.cc:290] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:01.373826 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 9a1031d785b646b095ece9be666d2a77 (DEFAULT_TABLE table=default.Table_a1d09325af844d71b290d95b8a4aaf12 [id=b62ec7bcad7c41b1b946e9ebf2f4bf39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.380558 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 50cea450eda9485daba40bf875a1b90c (DEFAULT_TABLE table=default.Table_3bd0db5c2861405a9dd9d66731cd9a57 [id=bfd41da632b94dcf90b0e0d230dc8fb5]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:01.380872 1444 leader_election.cc:343] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 3caecdff61ce48f7b7c3846bb380d21e
I20251024 19:48:01.381034 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 3e6db58fac5e4af9b27bde6c223aa845 (DEFAULT_TABLE table=default.Table_78e6806b2ede4763843892de1ab015e3 [id=1ac4b7fed3f145fc9ef5268266aeb9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.378012 1776 raft_consensus.cc:2243] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.383234 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee68243b4a664575b679950603bb250e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.383322 1776 raft_consensus.cc:2272] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.383733 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e6db58fac5e4af9b27bde6c223aa845. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.383805 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d01572e51694d9bb58c3a59d4205102. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.388026 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 3e6db58fac5e4af9b27bde6c223aa845 (DEFAULT_TABLE table=default.Table_78e6806b2ede4763843892de1ab015e3 [id=1ac4b7fed3f145fc9ef5268266aeb9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.389271 2098 raft_consensus.cc:2749] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:01.390805 1342 tablet_service.cc:1505] Processing CreateTablet for tablet ee68243b4a664575b679950603bb250e (DEFAULT_TABLE table=default.Table_605117e2eb734e6b8dc34ceb4b5b5b71 [id=38b6e1273e1a43bd8b12604db5371677]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.391264 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee68243b4a664575b679950603bb250e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.391695 1776 ts_tablet_manager.cc:1916] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.392349 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a1031d785b646b095ece9be666d2a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.392838 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 3e6db58fac5e4af9b27bde6c223aa845 (DEFAULT_TABLE table=default.Table_78e6806b2ede4763843892de1ab015e3 [id=1ac4b7fed3f145fc9ef5268266aeb9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.393429 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e6db58fac5e4af9b27bde6c223aa845. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:01.401964 1447 leader_election.cc:343] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 3caecdff61ce48f7b7c3846bb380d21e
I20251024 19:48:01.402168 1447 leader_election.cc:304] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:01.402786 2062 raft_consensus.cc:2749] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:01.403524 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 129debb55f4a4b7295a8ad88e55ebc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.404753 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d01572e51694d9bb58c3a59d4205102. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.405812 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c92b080805845c6af8f1cb160110915. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.406906 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6424b2347ef642feb38a9596f2d2be1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.408031 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf4596b333244cc9e9e35a128721e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.408921 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50cea450eda9485daba40bf875a1b90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.410435 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee68243b4a664575b679950603bb250e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.411018 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e6db58fac5e4af9b27bde6c223aa845. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.414028 1775 tablet_replica.cc:333] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:01.414354 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 6f21ffa44d7f48d08ed17ebf6c8ff29c (table default.Table_656be68a26b545d2b254abc2edb15254 [id=a1c4569aa6934d4a99787cd9e3240edb]) successfully deleted
I20251024 19:48:01.417829 1775 raft_consensus.cc:2243] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.418880 1775 raft_consensus.cc:2272] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.423709 1775 ts_tablet_manager.cc:1916] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.417325 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 50cea450eda9485daba40bf875a1b90c (DEFAULT_TABLE table=default.Table_3bd0db5c2861405a9dd9d66731cd9a57 [id=bfd41da632b94dcf90b0e0d230dc8fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.427551 1776 ts_tablet_manager.cc:1929] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.429709 1776 log.cc:1199] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/cc0bf7aee1a1421b9d985020a1674639
I20251024 19:48:01.429800 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2ea545b2cc14f589eda9419e55234bf" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:01.430552 1510 raft_consensus.cc:3060] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:01.434067 1776 ts_tablet_manager.cc:1950] T cc0bf7aee1a1421b9d985020a1674639 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:01.437399 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet cc0bf7aee1a1421b9d985020a1674639 (table default.Table_272323b2c4ac4c8cb0ff412a632ee43d [id=48bed8c74a444f9fa83d52387a2650c5]) successfully deleted
I20251024 19:48:01.439359 1776 tablet_replica.cc:333] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:01.439683 1776 raft_consensus.cc:2243] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.439926 1776 raft_consensus.cc:2272] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.440898 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 187bc968c2b049028dd25dfb0264c0aa
I20251024 19:48:01.441103 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 639836c656f44a819036f4d00e055bd9
I20251024 19:48:01.443344 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 59c7dbe9ff2c4869994c4f43d5676c30
I20251024 19:48:01.444219 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9489a01434a24d9ca94791252950725c
I20251024 19:48:01.445844 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 187bc968c2b049028dd25dfb0264c0aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:01.446467 1776 ts_tablet_manager.cc:1916] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.446748 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_36fd0e9f25584f9293267ff08c700997"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:01.447006 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 59c7dbe9ff2c4869994c4f43d5676c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.447242 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 639836c656f44a819036f4d00e055bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.447258 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 9489a01434a24d9ca94791252950725c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:01.447556 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36fd0e9f25584f9293267ff08c700997 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.448216 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50cea450eda9485daba40bf875a1b90c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.448935 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_83162e45d21e4d47a1cb086b20db8ef3" } new_table_name: "default.Table_4aaaf3ef7e68447f8e9c6576c7405f5a" modify_external_catalogs: true
I20251024 19:48:01.449100 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_bf08a3652bf34e8384edcf3d343795a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.449644 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf08a3652bf34e8384edcf3d343795a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.452227 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_719145aefd754122ba1bb0476ca4ac7d" } new_table_name: "default.Table_814a6dad46d54945b70e652b7e0f0032" modify_external_catalogs: true
I20251024 19:48:01.453560 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8c5e539b72dd4150adca95d805acce44" } new_table_name: "default.Table_b905a1285caa407e9fb08e81639c9bc8" modify_external_catalogs: true
I20251024 19:48:01.459012 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_19bbc037a7f241f5b338f4cfa4ebe82d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:01.459381 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_74e6a6d8b7a34049994d41a379a9a9f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.459734 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19bbc037a7f241f5b338f4cfa4ebe82d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.459728 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 59c7dbe9ff2c4869994c4f43d5676c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:01.459875 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74e6a6d8b7a34049994d41a379a9a9f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.460065 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet 9489a01434a24d9ca94791252950725c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:01.462148 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5a60ae2f44c2462086e73f80ad013457" } new_table_name: "default.Table_9abaf6cefac3462491207156404d36b7" modify_external_catalogs: true
I20251024 19:48:01.464090 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d7a9ef5436984a1b83dbeac77c302549"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:01.464175 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 639836c656f44a819036f4d00e055bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:01.464619 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7a9ef5436984a1b83dbeac77c302549 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.465350 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 187bc968c2b049028dd25dfb0264c0aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.474977 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e6ec362938cc478d9b9abee36f34f5c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.475610 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6ec362938cc478d9b9abee36f34f5c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:01.481766 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2602335327fc489fa154056c5ef5138f consensus state has no leader
I20251024 19:48:01.482077 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet 758df87acbdb4adaa83e1ecbc44e7cf1 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:01.484941 1776 ts_tablet_manager.cc:1929] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.490541 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3fefd748cb624ce2bad33072fbbb3aec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.491297 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fefd748cb624ce2bad33072fbbb3aec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.492184 1768 tablet_bootstrap.cc:492] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:01.492408 1768 ts_tablet_manager.cc:1403] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.441s user 0.052s sys 0.108s
I20251024 19:48:01.465227 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 59c7dbe9ff2c4869994c4f43d5676c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.465691 1480 tablet_service.cc:1552] Processing DeleteTablet for tablet 187bc968c2b049028dd25dfb0264c0aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.465940 1481 tablet_service.cc:1552] Processing DeleteTablet for tablet 639836c656f44a819036f4d00e055bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.466054 2020 raft_consensus.cc:493] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.495419 2020 raft_consensus.cc:515] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:01.496207 2020 leader_election.cc:290] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:01.465709 1478 tablet_service.cc:1552] Processing DeleteTablet for tablet 9489a01434a24d9ca94791252950725c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:01.497181 1179 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d38154d3ffd34f4b8d7305b1c23a4fae (table default.Table_c570270b7d7b4f5091c34e0550dd46ef [id=ed879934f5754b92b8cff234f3f66333], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet d38154d3ffd34f4b8d7305b1c23a4fae consensus state has no leader
I20251024 19:48:01.498658 1776 log.cc:1199] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/2e956ea914c545d79be6d632bfc4ec57
W20251024 19:48:01.499002 1447 leader_election.cc:343] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 2e956ea914c545d79be6d632bfc4ec57
I20251024 19:48:01.499174 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_646390b8820e4af996e1fa10cf39ac39" } new_table_name: "default.Table_18cdd8018c304deba24dd3429fdb48fc" modify_external_catalogs: true
W20251024 19:48:01.500797 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 067631b128fd4a1e9ab72c9507dab6c4 consensus state has no leader
I20251024 19:48:01.501220 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c7ecd114a8d44f5da18755c9a5ebed07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.501873 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7ecd114a8d44f5da18755c9a5ebed07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.503561 2048 raft_consensus.cc:493] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.506170 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e956ea914c545d79be6d632bfc4ec57" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:01.506451 1776 ts_tablet_manager.cc:1950] T 2e956ea914c545d79be6d632bfc4ec57 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
W20251024 19:48:01.506841 1444 leader_election.cc:343] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: Tablet not RUNNING: STOPPED
I20251024 19:48:01.506994 1444 leader_election.cc:304] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:01.507427 2020 raft_consensus.cc:2749] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:01.507630 2048 raft_consensus.cc:515] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.508244 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 2e956ea914c545d79be6d632bfc4ec57 (table default.Table_3819751f36bf419e9edd0f1ff339713a [id=350b37bf42194fcfbb660bffe83e47c4]) successfully deleted
W20251024 19:48:01.508579 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 5b96fc2309274048bed7a35e4bf4bae2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b96fc2309274048bed7a35e4bf4bae2 already in progress: creating tablet
W20251024 19:48:01.508905 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 16bc737104ab46b3b071a0760dbc847d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16bc737104ab46b3b071a0760dbc847d already in progress: creating tablet
I20251024 19:48:01.509130 1776 tablet_replica.cc:333] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:01.509328 1776 raft_consensus.cc:2243] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.510041 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9820e68ad5264df1a79b22f3b5369c5a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:01.514004 2098 raft_consensus.cc:493] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.517071 2048 leader_election.cc:290] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:01.517074 1776 raft_consensus.cc:2272] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:01.517299 1586 leader_election.cc:343] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:01.518208 1586 leader_election.cc:343] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 9820e68ad5264df1a79b22f3b5369c5a
I20251024 19:48:01.518332 1586 leader_election.cc:304] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:01.518606 1510 raft_consensus.cc:2468] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:01.520494 2109 tablet.cc:1722] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:01.520929 2048 raft_consensus.cc:2749] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:01.524580 1766 tablet_bootstrap.cc:492] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:01.524742 1766 ts_tablet_manager.cc:1403] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.328s user 0.007s sys 0.004s
I20251024 19:48:01.525512 1766 raft_consensus.cc:359] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.525723 1766 raft_consensus.cc:385] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.525816 1766 raft_consensus.cc:740] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.526535 1776 ts_tablet_manager.cc:1916] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.530000 1775 ts_tablet_manager.cc:1929] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.530145 1775 log.cc:1199] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/230606a0ce084b7bab0e41e48cf8fa89
I20251024 19:48:01.530463 1775 ts_tablet_manager.cc:1950] T 230606a0ce084b7bab0e41e48cf8fa89 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:01.531423 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 230606a0ce084b7bab0e41e48cf8fa89 (table default.Table_ebef5e3474bb4a288fcbaf6def6a7ce1 [id=469c675d15e3482d98671ee6b9e4bd42]) successfully deleted
I20251024 19:48:01.533004 1766 consensus_queue.cc:260] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.533766 2098 raft_consensus.cc:515] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.534600 2098 leader_election.cc:290] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:01.535180 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3b4e85f741849b2a6a3e8574396ff88 consensus state has no leader
I20251024 19:48:01.535436 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9623d7baf0174a019e2353a5effef081" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:01.535398 1768 raft_consensus.cc:359] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:01.536007 1305 leader_election.cc:343] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:01.535673 1768 raft_consensus.cc:385] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.535248 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9623d7baf0174a019e2353a5effef081" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:01.536336 1768 raft_consensus.cc:740] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.537032 1766 ts_tablet_manager.cc:1434] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251024 19:48:01.537189 1185 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 8ba85164015448d4a688a473cce90c5c (table default.Table_9764ce0360fd4fcfb03ab7df33ed112a [id=0fa321e2d4d847e69435f6de429d8779]). Expected version 1 got 0
W20251024 19:48:01.536609 1308 leader_election.cc:343] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:01.537636 1308 leader_election.cc:304] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:01.537927 1766 tablet_bootstrap.cc:492] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:01.538122 2098 raft_consensus.cc:2749] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:01.538525 1768 consensus_queue.cc:260] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:01.539162 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet dc1b6f0a377945089c50012f74d38564 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:01.539708 1768 ts_tablet_manager.cc:1434] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.047s user 0.001s sys 0.001s
I20251024 19:48:01.540009 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_40abb33fa043417bb208476294761da9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:01.540174 1768 tablet_bootstrap.cc:492] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:01.540267 1774 ts_tablet_manager.cc:1929] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:01.540405 1774 log.cc:1199] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/047436af5f6d4f688bd83e33089d3ee4
W20251024 19:48:01.540616 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40abb33fa043417bb208476294761da9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.540907 1774 ts_tablet_manager.cc:1950] T 047436af5f6d4f688bd83e33089d3ee4 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:01.541240 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 177) took 2738 ms. Trace:
I20251024 19:48:01.541380 1774 rpcz_store.cc:276] 1024 19:47:58.802405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.802463 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:01.320817 (+2518354us) ts_tablet_manager.cc:1151] Deleting tablet 047436af5f6d4f688bd83e33089d3ee4
1024 19:48:01.320827 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:01.490495 (+169668us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:01.503333 (+ 12838us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:01.541199 (+ 37866us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70,"tablet-delete.queue_time_us":2517948}
I20251024 19:48:01.541761 1766 tablet_bootstrap.cc:654] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:01.542217 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f89a3bbeb9b4d57a49078729110d951 consensus state has no leader
I20251024 19:48:01.542591 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_06b07944861041f8a66001a6555d7659" } modify_external_catalogs: true
I20251024 19:48:01.542755 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_06b07944861041f8a66001a6555d7659" } modify_external_catalogs: true
I20251024 19:48:01.543565 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_628b2afa9401426980be0eae5dffd917"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:01.544106 1768 tablet_bootstrap.cc:654] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:01.544092 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_628b2afa9401426980be0eae5dffd917 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:01.546092 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f already in progress: creating tablet
I20251024 19:48:01.546437 1775 tablet_replica.cc:333] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:01.546631 1775 raft_consensus.cc:2243] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.546829 1775 raft_consensus.cc:2272] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.547413 1775 ts_tablet_manager.cc:1916] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.547626 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (DEFAULT_TABLE table=default.Table_5f6d82174b9d4f64bbf61f72e5b58a25 [id=14bda7f9324f44aa9cb87a5f129039be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.548111 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10891ff5bf0d4b93bcab9faa5f7346f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.549908 1586 leader_election.cc:304] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
W20251024 19:48:01.550148 1586 raft_consensus.cc:2686] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251024 19:48:01.550652 1624 tablet_service.cc:1505] Processing CreateTablet for tablet d840888183ec4ef7bb245363c5bac322 (DEFAULT_TABLE table=default.Table_cb99a44a90e849be948f4a481c07a91d [id=5edc315e5e43438f83c57c449df681a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.551164 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d840888183ec4ef7bb245363c5bac322. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.552903 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (DEFAULT_TABLE table=default.Table_5f6d82174b9d4f64bbf61f72e5b58a25 [id=14bda7f9324f44aa9cb87a5f129039be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.553342 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10891ff5bf0d4b93bcab9faa5f7346f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.553898 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 59956dad06064024ab171631ef6c1dca (DEFAULT_TABLE table=default.Table_a5cf2d40da674fc592fd529d58dc0575 [id=e51bd6230e2e41429af7e8b08198bbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.554358 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59956dad06064024ab171631ef6c1dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.557936 1486 tablet_service.cc:1505] Processing CreateTablet for tablet d840888183ec4ef7bb245363c5bac322 (DEFAULT_TABLE table=default.Table_cb99a44a90e849be948f4a481c07a91d [id=5edc315e5e43438f83c57c449df681a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.558048 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 1edc55caab904722bf19a1c71df601cf (DEFAULT_TABLE table=default.Table_641c979b063f419884a09e448c68a1b4 [id=b72bff5d78ec4eb08810442a084a16a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.558362 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d840888183ec4ef7bb245363c5bac322. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.558563 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1edc55caab904722bf19a1c71df601cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.561968 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 59956dad06064024ab171631ef6c1dca (DEFAULT_TABLE table=default.Table_a5cf2d40da674fc592fd529d58dc0575 [id=e51bd6230e2e41429af7e8b08198bbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.562465 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59956dad06064024ab171631ef6c1dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.562598 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (DEFAULT_TABLE table=default.Table_5f6d82174b9d4f64bbf61f72e5b58a25 [id=14bda7f9324f44aa9cb87a5f129039be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.563872 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10891ff5bf0d4b93bcab9faa5f7346f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.566125 1349 tablet_service.cc:1505] Processing CreateTablet for tablet d840888183ec4ef7bb245363c5bac322 (DEFAULT_TABLE table=default.Table_cb99a44a90e849be948f4a481c07a91d [id=5edc315e5e43438f83c57c449df681a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.567025 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d840888183ec4ef7bb245363c5bac322. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.567260 1478 tablet_service.cc:1505] Processing CreateTablet for tablet 1edc55caab904722bf19a1c71df601cf (DEFAULT_TABLE table=default.Table_641c979b063f419884a09e448c68a1b4 [id=b72bff5d78ec4eb08810442a084a16a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.567883 1478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1edc55caab904722bf19a1c71df601cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.570029 1480 tablet_service.cc:1505] Processing CreateTablet for tablet 9026d9e801294dd2bfdb9c90e45a0990 (DEFAULT_TABLE table=default.Table_2952ae2bfd4d4924afd95ddf2e8404bd [id=1841054cd03e4989a236ebf134fa7755]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.570433 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9026d9e801294dd2bfdb9c90e45a0990. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.570849 1767 tablet_bootstrap.cc:492] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:01.571059 1767 ts_tablet_manager.cc:1403] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.351s user 0.052s sys 0.070s
I20251024 19:48:01.572054 1767 raft_consensus.cc:359] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.572283 1767 raft_consensus.cc:385] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.572377 1767 raft_consensus.cc:740] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.573865 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e956ea914c545d79be6d632bfc4ec57 with delete_type TABLET_DATA_DELETED (replaced by tablet a3259823b622461fa823841925b3d203) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.574209 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ea545b2cc14f589eda9419e55234bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.567255 1481 tablet_service.cc:1505] Processing CreateTablet for tablet a1fbbba88a294adfbf97c0343d86698e (DEFAULT_TABLE table=default.Table_8d1e5bb0a09942ac81957d503c4b78b2 [id=a60c8222aa0248c091fab8dc34100847]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.575407 1481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1fbbba88a294adfbf97c0343d86698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.575407 1630 tablet_service.cc:1505] Processing CreateTablet for tablet d2a68bb1dc6c483388784b9bb19ce065 (DEFAULT_TABLE table=default.Table_3be1c2c5954947969e6903a3f016d589 [id=0b304e4408b24fe797f4438b874e480f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.575829 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a68bb1dc6c483388784b9bb19ce065. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.577670 1621 tablet_service.cc:1505] Processing CreateTablet for tablet f965111ac60841dbb9e6ecc21ff3fec0 (DEFAULT_TABLE table=default.Table_d696d859aaad48ceac033959d75bcf11 [id=c727179cd01648cd86e1d9eca2396cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.577874 1622 tablet_service.cc:1505] Processing CreateTablet for tablet a1fbbba88a294adfbf97c0343d86698e (DEFAULT_TABLE table=default.Table_8d1e5bb0a09942ac81957d503c4b78b2 [id=a60c8222aa0248c091fab8dc34100847]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.578051 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f965111ac60841dbb9e6ecc21ff3fec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.578218 1485 tablet_service.cc:1505] Processing CreateTablet for tablet d2a68bb1dc6c483388784b9bb19ce065 (DEFAULT_TABLE table=default.Table_3be1c2c5954947969e6903a3f016d589 [id=0b304e4408b24fe797f4438b874e480f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.578295 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1fbbba88a294adfbf97c0343d86698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.578583 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a68bb1dc6c483388784b9bb19ce065. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.581903 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 9026d9e801294dd2bfdb9c90e45a0990 (DEFAULT_TABLE table=default.Table_2952ae2bfd4d4924afd95ddf2e8404bd [id=1841054cd03e4989a236ebf134fa7755]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.582296 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9026d9e801294dd2bfdb9c90e45a0990. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.583195 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 385e71e672154e399f14321eac0d8625 (DEFAULT_TABLE table=default.Table_775e17f58fef4f2b850035a03e411a08 [id=6239ceae506442b79bbe79c48813b49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.583578 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385e71e672154e399f14321eac0d8625. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.577894 1611 tablet_service.cc:1505] Processing CreateTablet for tablet f3d80885fcc642b8a75da30cd4abecbd (DEFAULT_TABLE table=default.Table_7966ff9879d24254aed39562b2f127d7 [id=45b38d5e9fd6472fa318b41863080708]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.584447 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 178) took 2687 ms. Trace:
I20251024 19:48:01.585256 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 1edc55caab904722bf19a1c71df601cf (DEFAULT_TABLE table=default.Table_641c979b063f419884a09e448c68a1b4 [id=b72bff5d78ec4eb08810442a084a16a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.585783 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3d80885fcc642b8a75da30cd4abecbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.586061 1491 tablet_service.cc:1505] Processing CreateTablet for tablet f3d80885fcc642b8a75da30cd4abecbd (DEFAULT_TABLE table=default.Table_7966ff9879d24254aed39562b2f127d7 [id=45b38d5e9fd6472fa318b41863080708]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.586645 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3d80885fcc642b8a75da30cd4abecbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.587024 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 28005301f9de48e5826f46c79aeb5fe5 (DEFAULT_TABLE table=default.Table_09c1a7fb5d6e47da800d84d81b233bce [id=e345af66b89e48f9afe96000ce50179c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.587370 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28005301f9de48e5826f46c79aeb5fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.587368 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1edc55caab904722bf19a1c71df601cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.591558 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 28005301f9de48e5826f46c79aeb5fe5 (DEFAULT_TABLE table=default.Table_09c1a7fb5d6e47da800d84d81b233bce [id=e345af66b89e48f9afe96000ce50179c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.591799 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 59956dad06064024ab171631ef6c1dca (DEFAULT_TABLE table=default.Table_a5cf2d40da674fc592fd529d58dc0575 [id=e51bd6230e2e41429af7e8b08198bbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.591975 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28005301f9de48e5826f46c79aeb5fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.592192 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59956dad06064024ab171631ef6c1dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.594233 1346 tablet_service.cc:1505] Processing CreateTablet for tablet a1fbbba88a294adfbf97c0343d86698e (DEFAULT_TABLE table=default.Table_8d1e5bb0a09942ac81957d503c4b78b2 [id=a60c8222aa0248c091fab8dc34100847]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.594408 1348 tablet_service.cc:1505] Processing CreateTablet for tablet d2a68bb1dc6c483388784b9bb19ce065 (DEFAULT_TABLE table=default.Table_3be1c2c5954947969e6903a3f016d589 [id=0b304e4408b24fe797f4438b874e480f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.594521 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 979dd87a8c66480ea314fcc75fa878c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.597837 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a68bb1dc6c483388784b9bb19ce065. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:01.600250 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4193750331874d7ba9bbe6279c3a8e37 consensus state has no leader
I20251024 19:48:01.601430 1482 tablet_service.cc:1505] Processing CreateTablet for tablet f965111ac60841dbb9e6ecc21ff3fec0 (DEFAULT_TABLE table=default.Table_d696d859aaad48ceac033959d75bcf11 [id=c727179cd01648cd86e1d9eca2396cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.603951 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1fbbba88a294adfbf97c0343d86698e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.604445 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 385e71e672154e399f14321eac0d8625 (DEFAULT_TABLE table=default.Table_775e17f58fef4f2b850035a03e411a08 [id=6239ceae506442b79bbe79c48813b49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.604847 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385e71e672154e399f14321eac0d8625. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.604645 2020 raft_consensus.cc:493] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.607343 2020 raft_consensus.cc:515] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.608285 2020 leader_election.cc:290] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:01.609943 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f965111ac60841dbb9e6ecc21ff3fec0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:01.611276 1444 leader_election.cc:343] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 8721ee81e4e24153a0aad8ce99f872e1
I20251024 19:48:01.625641 1767 consensus_queue.cc:260] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.628491 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 9026d9e801294dd2bfdb9c90e45a0990 (DEFAULT_TABLE table=default.Table_2952ae2bfd4d4924afd95ddf2e8404bd [id=1841054cd03e4989a236ebf134fa7755]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.628947 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9026d9e801294dd2bfdb9c90e45a0990. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.630137 1767 ts_tablet_manager.cc:1434] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.059s user 0.000s sys 0.003s
I20251024 19:48:01.630599 1767 tablet_bootstrap.cc:492] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:01.631557 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 28005301f9de48e5826f46c79aeb5fe5 (DEFAULT_TABLE table=default.Table_09c1a7fb5d6e47da800d84d81b233bce [id=e345af66b89e48f9afe96000ce50179c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.632088 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 385e71e672154e399f14321eac0d8625 (DEFAULT_TABLE table=default.Table_775e17f58fef4f2b850035a03e411a08 [id=6239ceae506442b79bbe79c48813b49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.632404 1347 tablet_service.cc:1505] Processing CreateTablet for tablet f3d80885fcc642b8a75da30cd4abecbd (DEFAULT_TABLE table=default.Table_7966ff9879d24254aed39562b2f127d7 [id=45b38d5e9fd6472fa318b41863080708]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.632642 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 047436af5f6d4f688bd83e33089d3ee4 (table default.Table_812ae4fbc0fc4fdea1317e0d9794c50b [id=93f77b3367c04c9bb7e3e02cfa050c36]) successfully deleted
I20251024 19:48:01.634091 1696 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
W20251024 19:48:01.634409 1447 leader_election.cc:343] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 8721ee81e4e24153a0aad8ce99f872e1
I20251024 19:48:01.634555 1447 leader_election.cc:304] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:01.635047 2020 raft_consensus.cc:2749] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:01.636992 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28005301f9de48e5826f46c79aeb5fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.638542 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3d80885fcc642b8a75da30cd4abecbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.644393 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385e71e672154e399f14321eac0d8625. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.646535 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 758df87acbdb4adaa83e1ecbc44e7cf1
I20251024 19:48:01.647102 1188 catalog_manager.cc:6719] ReplaceTablet: tablet 758df87acbdb4adaa83e1ecbc44e7cf1 deleted and replaced by tablet 3fd8aa1d9aca429dbfe46f472ebbf53b
I20251024 19:48:01.650434 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a5d11f93359546078ef5becf10539131" } modify_external_catalogs: true
I20251024 19:48:01.650642 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a5d11f93359546078ef5becf10539131" } modify_external_catalogs: true
I20251024 19:48:01.651475 1767 tablet_bootstrap.cc:654] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.652185 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3be1c2c5954947969e6903a3f016d589" } modify_external_catalogs: true
I20251024 19:48:01.652400 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3be1c2c5954947969e6903a3f016d589" } modify_external_catalogs: true
I20251024 19:48:01.654085 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_73923b0648674b2891cbbf4b90d3f106" } modify_external_catalogs: true
I20251024 19:48:01.654270 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_73923b0648674b2891cbbf4b90d3f106" } modify_external_catalogs: true
I20251024 19:48:01.655081 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fd8aa1d9aca429dbfe46f472ebbf53b) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:01.655790 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_641c979b063f419884a09e448c68a1b4" } modify_external_catalogs: true
I20251024 19:48:01.655962 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_641c979b063f419884a09e448c68a1b4" } modify_external_catalogs: true
I20251024 19:48:01.656425 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fd8aa1d9aca429dbfe46f472ebbf53b) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.657748 1776 ts_tablet_manager.cc:1929] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.658059 1776 log.cc:1199] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/b2ea545b2cc14f589eda9419e55234bf
I20251024 19:48:01.658807 1776 ts_tablet_manager.cc:1950] T b2ea545b2cc14f589eda9419e55234bf P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:01.661707 1776 tablet_replica.cc:333] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:01.662875 1766 tablet_bootstrap.cc:492] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:01.663108 1766 ts_tablet_manager.cc:1403] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.125s user 0.000s sys 0.003s
I20251024 19:48:01.664014 1766 raft_consensus.cc:359] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.664286 1766 raft_consensus.cc:385] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.664345 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet b2ea545b2cc14f589eda9419e55234bf (table default.Table_74349eed6d4b4e8e95ae01909bbb91e9 [id=94b8daec91b6478f9dc43734542246e0]) successfully deleted
I20251024 19:48:01.664402 1766 raft_consensus.cc:740] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.664671 1766 consensus_queue.cc:260] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.665283 1766 ts_tablet_manager.cc:1434] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:01.665491 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet dc1b6f0a377945089c50012f74d38564
I20251024 19:48:01.665987 1191 catalog_manager.cc:6719] ReplaceTablet: tablet dc1b6f0a377945089c50012f74d38564 deleted and replaced by tablet 6ca8f818440f4ae7a63c0901fcdcdf59
I20251024 19:48:01.667258 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet dc1b6f0a377945089c50012f74d38564 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ca8f818440f4ae7a63c0901fcdcdf59) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:01.667614 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet dc1b6f0a377945089c50012f74d38564 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ca8f818440f4ae7a63c0901fcdcdf59) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.665727 1766 tablet_bootstrap.cc:492] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:01.669950 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 9820e68ad5264df1a79b22f3b5369c5a with delete_type TABLET_DATA_DELETED (replaced by tablet e5f58c190a9a4c5391f1330f79865589) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.682148 1766 tablet_bootstrap.cc:654] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.686705 2020 raft_consensus.cc:493] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.686894 2020 raft_consensus.cc:515] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.687613 2020 leader_election.cc:290] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:01.689482 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 62667092d7e244cca1020ec220ea9124
I20251024 19:48:01.690223 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 62667092d7e244cca1020ec220ea9124 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:01.690524 1447 leader_election.cc:343] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 15c08dcf379f4df9a0e461d6dcfeeac4
I20251024 19:48:01.692525 1766 tablet_bootstrap.cc:492] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:01.692680 1766 ts_tablet_manager.cc:1403] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.003s
I20251024 19:48:01.693248 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8d1e5bb0a09942ac81957d503c4b78b2" } new_table_name: "default.Table_4c3e151bdb364b37ac92d4c3b767dd35" modify_external_catalogs: true
I20251024 19:48:01.693612 1766 raft_consensus.cc:359] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.693797 1766 raft_consensus.cc:385] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.693892 1766 raft_consensus.cc:740] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.694151 1766 consensus_queue.cc:260] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.696755 1766 ts_tablet_manager.cc:1434] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251024 19:48:01.697113 1766 tablet_bootstrap.cc:492] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:01.697306 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_81453b8401974cbcaa50e97a3c3bf12d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.697928 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81453b8401974cbcaa50e97a3c3bf12d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.699796 1766 tablet_bootstrap.cc:654] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.701664 1557 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:01.702486 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet f91316c030dc4f7da27977a9dde01f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.702382 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2c02fd9cab9b479d8876a6dcbd586b17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.702960 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c02fd9cab9b479d8876a6dcbd586b17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.705153 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_18392df3579642bf80ac127778fd508f" } new_table_name: "default.Table_dd245479047544d68efcf1ec83acc6b8" modify_external_catalogs: true
I20251024 19:48:01.706092 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 62667092d7e244cca1020ec220ea9124 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:01.708369 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 90405a0c13bd4c229b011c90232f751d with delete_type TABLET_DATA_DELETED (replaced by tablet 59c7dbe9ff2c4869994c4f43d5676c30) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.709534 1766 tablet_bootstrap.cc:492] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:01.709844 1766 ts_tablet_manager.cc:1403] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20251024 19:48:01.710881 1766 raft_consensus.cc:359] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.711161 1766 raft_consensus.cc:385] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.711292 1766 raft_consensus.cc:740] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.711593 1766 consensus_queue.cc:260] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.714185 1766 ts_tablet_manager.cc:1434] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:01.714540 1766 tablet_bootstrap.cc:492] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:01.715624 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet fe97e31310ae4733ba93de1b62a37842 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e6db58fac5e4af9b27bde6c223aa845) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.716171 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 6f630fd412ca4c4f8570359c1678e6af
I20251024 19:48:01.716450 1766 tablet_bootstrap.cc:654] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.717473 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f630fd412ca4c4f8570359c1678e6af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.731225 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet d2a68bb1dc6c483388784b9bb19ce065
I20251024 19:48:01.733009 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet d2a68bb1dc6c483388784b9bb19ce065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.733620 1345 tablet_service.cc:1505] Processing CreateTablet for tablet f965111ac60841dbb9e6ecc21ff3fec0 (DEFAULT_TABLE table=default.Table_d696d859aaad48ceac033959d75bcf11 [id=c727179cd01648cd86e1d9eca2396cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.734169 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f965111ac60841dbb9e6ecc21ff3fec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.739156 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f630fd412ca4c4f8570359c1678e6af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:01.739392 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet d2a68bb1dc6c483388784b9bb19ce065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:01.743453 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet fdfa5eea3b674a3389a156d1c72204df
I20251024 19:48:01.744196 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet fdfa5eea3b674a3389a156d1c72204df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.745128 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet fdfa5eea3b674a3389a156d1c72204df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:01.584666 1774 rpcz_store.cc:276] 1024 19:47:58.896685 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.905275 (+ 8590us) service_pool.cc:225] Handling call
1024 19:48:01.584089 (+2678814us) ts_tablet_manager.cc:1151] Deleting tablet 3754540ae21e4c65b5af3d0aa3f9189f
1024 19:48:01.584101 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:01.584346 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2678503}
W20251024 19:48:01.749933 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3754540ae21e4c65b5af3d0aa3f9189f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3754540ae21e4c65b5af3d0aa3f9189f already in progress: creating tablet
I20251024 19:48:01.750070 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_82dcb5dcc93042b4a560b867c853f30d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:01.750548 1774 tablet_replica.cc:333] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
W20251024 19:48:01.750677 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82dcb5dcc93042b4a560b867c853f30d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.750824 1774 raft_consensus.cc:2243] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.751026 1774 raft_consensus.cc:2272] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.751775 1774 ts_tablet_manager.cc:1916] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.752144 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_34ef66bee6bd460b92cf30b469b3f179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.752696 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34ef66bee6bd460b92cf30b469b3f179 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.753738 1776 raft_consensus.cc:2243] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.749488 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c4882268578941ccaf2c28be3074fdd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.754868 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4882268578941ccaf2c28be3074fdd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.759531 1775 ts_tablet_manager.cc:1929] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251024 19:48:01.759905 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a1fbbba88a294adfbf97c0343d86698e consensus state has no leader
I20251024 19:48:01.765870 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dd9b9ef96622439ea016c3813fcbb81a" } new_table_name: "default.Table_8ae396f2397d425097a8f71cb1c9c910" modify_external_catalogs: true
I20251024 19:48:01.767349 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9edfe580f57e43919a206badbc0bbfd6" } new_table_name: "default.Table_a4b63032d2224a6d95765cb93137f4a2" modify_external_catalogs: true
I20251024 19:48:01.769011 1768 tablet_bootstrap.cc:492] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:01.769254 1768 ts_tablet_manager.cc:1403] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.229s user 0.002s sys 0.002s
I20251024 19:48:01.769326 1190 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]). Expected version 3 got 0
I20251024 19:48:01.770484 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet e3b4e85f741849b2a6a3e8574396ff88 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:01.770346 1768 raft_consensus.cc:359] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.770596 1768 raft_consensus.cc:385] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.770964 1768 raft_consensus.cc:740] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.773871 1768 consensus_queue.cc:260] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.774194 2101 tablet.cc:1722] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251024 19:48:01.778618 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet c82f60819e7a442bbc0cc04659e14e87 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:01.781512 1418 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:01.785874 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet dc1b6f0a377945089c50012f74d38564 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ca8f818440f4ae7a63c0901fcdcdf59) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.786265 1776 raft_consensus.cc:2272] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.787029 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fd8aa1d9aca429dbfe46f472ebbf53b) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.787791 1776 ts_tablet_manager.cc:1916] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.789829 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f630fd412ca4c4f8570359c1678e6af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.794816 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet 27b32126d04e480f8db7b2b20aba5873 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:01.795981 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet d2a68bb1dc6c483388784b9bb19ce065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.796557 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet fdfa5eea3b674a3389a156d1c72204df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.796567 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6a4dbd8e2b264bbca433946dfb1890b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.797317 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a4dbd8e2b264bbca433946dfb1890b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.797326 1768 ts_tablet_manager.cc:1434] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20251024 19:48:01.799559 1767 tablet_bootstrap.cc:492] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:01.799763 1767 ts_tablet_manager.cc:1403] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.169s user 0.007s sys 0.014s
I20251024 19:48:01.801681 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 62667092d7e244cca1020ec220ea9124 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.802345 1767 raft_consensus.cc:359] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.802546 1767 raft_consensus.cc:385] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.802702 1767 raft_consensus.cc:740] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.802978 1767 consensus_queue.cc:260] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.803685 1767 ts_tablet_manager.cc:1434] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20251024 19:48:01.803807 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7503b22d9e12481b9af96385b9d76e50 consensus state has no leader
I20251024 19:48:01.804025 1767 tablet_bootstrap.cc:492] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:01.804708 1768 tablet_bootstrap.cc:492] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:01.805998 1767 tablet_bootstrap.cc:654] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.810801 1768 tablet_bootstrap.cc:654] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.810756 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8255c10f050f4670a31e7c175a36cb62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.811429 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8255c10f050f4670a31e7c175a36cb62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.814373 1775 log.cc:1199] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/3caecdff61ce48f7b7c3846bb380d21e
W20251024 19:48:01.759815 1444 leader_election.cc:343] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 15c08dcf379f4df9a0e461d6dcfeeac4
I20251024 19:48:01.814682 1444 leader_election.cc:304] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:01.814862 1775 ts_tablet_manager.cc:1950] T 3caecdff61ce48f7b7c3846bb380d21e P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
W20251024 19:48:01.815079 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 55284us
I20251024 19:48:01.773061 2109 tablet.cc:1722] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251024 19:48:01.815814 2020 raft_consensus.cc:2749] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:01.815838 1776 ts_tablet_manager.cc:1929] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.815979 1776 log.cc:1199] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/979dd87a8c66480ea314fcc75fa878c3
I20251024 19:48:01.816965 1776 ts_tablet_manager.cc:1950] T 979dd87a8c66480ea314fcc75fa878c3 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:01.823267 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9110872a9b40450bb070d0c1db2d2b84" } new_table_name: "default.Table_e962c1d3850943a0a9e86f5882d1c518" modify_external_catalogs: true
I20251024 19:48:01.824041 1776 tablet_replica.cc:333] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:01.824304 1776 raft_consensus.cc:2243] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:01.824580 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 979dd87a8c66480ea314fcc75fa878c3 (table default.Table_e0d4700f7fd849399f95eb388d194091 [id=cf70cd6b11614a0d8b39b707311c46fd]) successfully deleted
I20251024 19:48:01.824973 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet b9b1e0338fd14634a1e921735ae61289 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.825433 1775 tablet_replica.cc:333] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:01.825413 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 3caecdff61ce48f7b7c3846bb380d21e (table default.Table_8c169b038d764be684a4e5b56faf745b [id=e3c15f9790304d4eb7be44889ef89d20]) successfully deleted
I20251024 19:48:01.825698 1775 raft_consensus.cc:2243] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:01.825995 1775 pending_rounds.cc:70] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e: Trying to abort 1 pending ops.
I20251024 19:48:01.826143 1775 pending_rounds.cc:77] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429311763156992 op_type: NO_OP noop_request { }
I20251024 19:48:01.826296 1775 raft_consensus.cc:2889] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:01.826431 1775 raft_consensus.cc:2272] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:01.830964 1775 ts_tablet_manager.cc:1916] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.831355 1776 raft_consensus.cc:2272] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:01.833824 2048 raft_consensus.cc:670] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20251024 19:48:01.834630 1776 ts_tablet_manager.cc:1916] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:01.843885 1767 tablet_bootstrap.cc:492] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:01.844211 1767 ts_tablet_manager.cc:1403] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.001s
I20251024 19:48:01.845088 1767 raft_consensus.cc:359] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.845397 1767 raft_consensus.cc:385] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.845546 1767 raft_consensus.cc:740] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.845877 1767 consensus_queue.cc:260] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.846454 1767 ts_tablet_manager.cc:1434] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:01.846810 1767 tablet_bootstrap.cc:492] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:01.850353 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ba7eebe9102b4e27ad8227e35ed754ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.850965 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba7eebe9102b4e27ad8227e35ed754ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.851279 2122 tablet.cc:1722] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251024 19:48:01.854245 1767 tablet_bootstrap.cc:654] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:01.856637 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f83b9290f00145a780318a43249eec12 consensus state has no leader
I20251024 19:48:01.860307 1776 ts_tablet_manager.cc:1929] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.862434 1766 tablet_bootstrap.cc:492] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:01.862680 1766 ts_tablet_manager.cc:1403] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.148s user 0.021s sys 0.014s
I20251024 19:48:01.863180 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_98de64dffcd34afc91c487faa22cb705"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.863929 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98de64dffcd34afc91c487faa22cb705 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.864302 1766 raft_consensus.cc:359] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.864543 1766 raft_consensus.cc:385] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.864727 1766 raft_consensus.cc:740] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.865052 1766 consensus_queue.cc:260] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.871584 1766 ts_tablet_manager.cc:1434] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251024 19:48:01.873013 1766 tablet_bootstrap.cc:492] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:01.875802 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d129bdc876e425aabcf1c99997e319a consensus state has no leader
I20251024 19:48:01.875895 1766 tablet_bootstrap.cc:654] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.881721 1774 ts_tablet_manager.cc:1929] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:01.882462 1768 tablet_bootstrap.cc:492] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:01.882834 1774 log.cc:1199] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/c292ce055def4964ab143252c6b2b87e
I20251024 19:48:01.883177 1774 ts_tablet_manager.cc:1950] T c292ce055def4964ab143252c6b2b87e P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:01.884094 1768 ts_tablet_manager.cc:1403] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.014s
I20251024 19:48:01.885052 1768 raft_consensus.cc:359] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:01.885234 1768 raft_consensus.cc:385] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.885309 1768 raft_consensus.cc:740] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.885493 1768 consensus_queue.cc:260] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:01.886042 1768 ts_tablet_manager.cc:1434] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:01.886376 1768 tablet_bootstrap.cc:492] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:01.887837 1775 ts_tablet_manager.cc:1929] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:01.887990 1775 log.cc:1199] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/cc0bf7aee1a1421b9d985020a1674639
I20251024 19:48:01.888527 1775 ts_tablet_manager.cc:1950] T cc0bf7aee1a1421b9d985020a1674639 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:01.889374 1768 tablet_bootstrap.cc:654] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.889791 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet cc0bf7aee1a1421b9d985020a1674639 (table default.Table_272323b2c4ac4c8cb0ff412a632ee43d [id=48bed8c74a444f9fa83d52387a2650c5]) successfully deleted
I20251024 19:48:01.892339 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 96a6e0a4882c42399fd07f06ac21aa3f (DEFAULT_TABLE table=default.Table_19bbc037a7f241f5b338f4cfa4ebe82d [id=b2ce46e067994c9fa4df9f00de59d68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.892813 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a6e0a4882c42399fd07f06ac21aa3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.893059 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 96a6e0a4882c42399fd07f06ac21aa3f (DEFAULT_TABLE table=default.Table_19bbc037a7f241f5b338f4cfa4ebe82d [id=b2ce46e067994c9fa4df9f00de59d68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.893456 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a6e0a4882c42399fd07f06ac21aa3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.900704 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 65424b00ab9c454f9584048caba83e6a (DEFAULT_TABLE table=default.Table_e6ec362938cc478d9b9abee36f34f5c5 [id=d2e2211a2cd8457aab95305acead0b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.901310 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 4eafde5a40df4fbfb03340b7bf791a2a (DEFAULT_TABLE table=default.Table_bf08a3652bf34e8384edcf3d343795a7 [id=efad7044899143ef90b4114a22500053]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.901800 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 38818c7997c84cedad5f9b09eef77105 (DEFAULT_TABLE table=default.Table_d7a9ef5436984a1b83dbeac77c302549 [id=4c4868b6a38046faaf24f65fd00420cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.902194 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 179) took 2958 ms. Trace:
I20251024 19:48:01.902308 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 250f106664be4b5aaa330dbb4e43cb7b (DEFAULT_TABLE table=default.Table_36fd0e9f25584f9293267ff08c700997 [id=963d13a1679345489335e30a9aa349b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.902842 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 1999b751e9344dcf81d26226c0589c8f (DEFAULT_TABLE table=default.Table_3fefd748cb624ce2bad33072fbbb3aec [id=d743705c21da4b89abedf2eb34ddbe09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.903344 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 0266cef4e2034303978d7456c47ea98c (DEFAULT_TABLE table=default.Table_74e6a6d8b7a34049994d41a379a9a9f0 [id=7f87fee0587e4cb5a4f6f5afb3238334]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.904814 1480 tablet_service.cc:1505] Processing CreateTablet for tablet 65424b00ab9c454f9584048caba83e6a (DEFAULT_TABLE table=default.Table_e6ec362938cc478d9b9abee36f34f5c5 [id=d2e2211a2cd8457aab95305acead0b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.905179 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65424b00ab9c454f9584048caba83e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.865419 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 369cebfda6ee43638b79f8652fa55855 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.865857 1767 tablet_bootstrap.cc:492] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:01.892259 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 96a6e0a4882c42399fd07f06ac21aa3f (DEFAULT_TABLE table=default.Table_19bbc037a7f241f5b338f4cfa4ebe82d [id=b2ce46e067994c9fa4df9f00de59d68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.906788 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a6e0a4882c42399fd07f06ac21aa3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.908275 1192 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9] alter complete (version 3)
I20251024 19:48:01.894587 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 65424b00ab9c454f9584048caba83e6a (DEFAULT_TABLE table=default.Table_e6ec362938cc478d9b9abee36f34f5c5 [id=d2e2211a2cd8457aab95305acead0b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.909792 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65424b00ab9c454f9584048caba83e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.909792 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet a48249f1824a40b087b80a563816d485 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:01.910377 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65424b00ab9c454f9584048caba83e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.913152 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ea7e190995344cb8a1c35c95fc35b820"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:01.895027 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 4eafde5a40df4fbfb03340b7bf791a2a (DEFAULT_TABLE table=default.Table_bf08a3652bf34e8384edcf3d343795a7 [id=efad7044899143ef90b4114a22500053]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.895730 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 38818c7997c84cedad5f9b09eef77105 (DEFAULT_TABLE table=default.Table_d7a9ef5436984a1b83dbeac77c302549 [id=4c4868b6a38046faaf24f65fd00420cb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:01.913796 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea7e190995344cb8a1c35c95fc35b820 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.896369 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 250f106664be4b5aaa330dbb4e43cb7b (DEFAULT_TABLE table=default.Table_36fd0e9f25584f9293267ff08c700997 [id=963d13a1679345489335e30a9aa349b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.914156 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250f106664be4b5aaa330dbb4e43cb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.914244 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38818c7997c84cedad5f9b09eef77105. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.914640 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eafde5a40df4fbfb03340b7bf791a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.915012 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1999b751e9344dcf81d26226c0589c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.915212 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_78e6806b2ede4763843892de1ab015e3" } modify_external_catalogs: true
I20251024 19:48:01.915400 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_78e6806b2ede4763843892de1ab015e3" } modify_external_catalogs: true
I20251024 19:48:01.915473 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250f106664be4b5aaa330dbb4e43cb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.915930 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0266cef4e2034303978d7456c47ea98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.916519 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 38818c7997c84cedad5f9b09eef77105 (DEFAULT_TABLE table=default.Table_d7a9ef5436984a1b83dbeac77c302549 [id=4c4868b6a38046faaf24f65fd00420cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.916994 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38818c7997c84cedad5f9b09eef77105. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.917891 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38818c7997c84cedad5f9b09eef77105. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.918061 1185 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 2346c03802a245a2af8c086b626d660d (table default.Table_48177e84ca024d49a2fd4e205a08a2a9 [id=aadca5a1b6d4494aa37f8a8cc76ad12a]). Expected version 4 got 0
I20251024 19:48:01.918774 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 4eafde5a40df4fbfb03340b7bf791a2a (DEFAULT_TABLE table=default.Table_bf08a3652bf34e8384edcf3d343795a7 [id=efad7044899143ef90b4114a22500053]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.919111 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eafde5a40df4fbfb03340b7bf791a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.919705 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_52ec55c23c394294a68850ab6b33aabe" } modify_external_catalogs: true
I20251024 19:48:01.919884 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_52ec55c23c394294a68850ab6b33aabe" } modify_external_catalogs: true
I20251024 19:48:01.921422 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eafde5a40df4fbfb03340b7bf791a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.921985 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 1999b751e9344dcf81d26226c0589c8f (DEFAULT_TABLE table=default.Table_3fefd748cb624ce2bad33072fbbb3aec [id=d743705c21da4b89abedf2eb34ddbe09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.922483 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1999b751e9344dcf81d26226c0589c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.922996 1478 tablet_service.cc:1505] Processing CreateTablet for tablet 0266cef4e2034303978d7456c47ea98c (DEFAULT_TABLE table=default.Table_74e6a6d8b7a34049994d41a379a9a9f0 [id=7f87fee0587e4cb5a4f6f5afb3238334]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.923611 1478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0266cef4e2034303978d7456c47ea98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.897375 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 1999b751e9344dcf81d26226c0589c8f (DEFAULT_TABLE table=default.Table_3fefd748cb624ce2bad33072fbbb3aec [id=d743705c21da4b89abedf2eb34ddbe09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.926002 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1999b751e9344dcf81d26226c0589c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.898092 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 0266cef4e2034303978d7456c47ea98c (DEFAULT_TABLE table=default.Table_74e6a6d8b7a34049994d41a379a9a9f0 [id=7f87fee0587e4cb5a4f6f5afb3238334]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.929639 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 250f106664be4b5aaa330dbb4e43cb7b (DEFAULT_TABLE table=default.Table_36fd0e9f25584f9293267ff08c700997 [id=963d13a1679345489335e30a9aa349b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:01.929977 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0266cef4e2034303978d7456c47ea98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.930521 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250f106664be4b5aaa330dbb4e43cb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:01.933339 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9ecacbf3e0fe41b7b54b20dbd81c4e05" } modify_external_catalogs: true
I20251024 19:48:01.933559 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9ecacbf3e0fe41b7b54b20dbd81c4e05" } modify_external_catalogs: true
I20251024 19:48:01.934213 1768 tablet_bootstrap.cc:492] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:01.934520 1768 ts_tablet_manager.cc:1403] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.006s
I20251024 19:48:01.935603 1768 raft_consensus.cc:359] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.935889 1768 raft_consensus.cc:385] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.935990 1768 raft_consensus.cc:740] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.936331 1768 consensus_queue.cc:260] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.941227 1768 ts_tablet_manager.cc:1434] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20251024 19:48:01.941681 1768 tablet_bootstrap.cc:492] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:01.943625 1768 tablet_bootstrap.cc:654] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:01.944118 1186 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet c04082bd1b444b468c177bef239701c6 (table default.Table_24200cd6f19d460cae963afb9b7f8724 [id=a2789faf4abe4c09b0a7515d941831a6]). Expected version 2 got 0
I20251024 19:48:01.945348 2020 raft_consensus.cc:493] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.945565 2020 raft_consensus.cc:515] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.946331 2020 leader_election.cc:290] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:01.946866 1481 tablet_service.cc:1552] Processing DeleteTablet for tablet f05ad092936b4bafa3ae795904c2df78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.947161 1186 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet c04082bd1b444b468c177bef239701c6 (table default.Table_24200cd6f19d460cae963afb9b7f8724 [id=a2789faf4abe4c09b0a7515d941831a6]). Expected version 2 got 0
I20251024 19:48:01.947831 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe97e31310ae4733ba93de1b62a37842" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:01.948758 2098 raft_consensus.cc:493] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:01.948843 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe97e31310ae4733ba93de1b62a37842" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:01.948982 2098 raft_consensus.cc:515] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:01.949090 1371 raft_consensus.cc:2468] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:01.949731 2098 leader_election.cc:290] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:01.950134 1444 leader_election.cc:343] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:01.950464 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e20881e400cf4fac914b70afc79c084f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:01.950541 1447 leader_election.cc:304] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:01.951035 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e20881e400cf4fac914b70afc79c084f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:01.951076 1308 leader_election.cc:343] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:01.951591 1305 leader_election.cc:343] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:01.952935 2020 raft_consensus.cc:2804] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:01.953053 2020 raft_consensus.cc:493] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:01.953137 2020 raft_consensus.cc:3060] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:01.955363 2057 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 13251701906526438863) tablet 00000000000000000000000000000000 cur holder 0x617000052480
W20251024 19:48:01.958194 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 607ae453221d422e9d07882830c02275 consensus state has no leader
I20251024 19:48:01.954288 1305 leader_election.cc:304] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:01.962327 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_19bbc037a7f241f5b338f4cfa4ebe82d" } modify_external_catalogs: true
I20251024 19:48:01.962599 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_19bbc037a7f241f5b338f4cfa4ebe82d" } modify_external_catalogs: true
I20251024 19:48:01.966354 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_31d09f80ce0a4953b4764b8425875f89" } new_table_name: "default.Table_fb1d11002e79489c89a631ed3d11a5c3" modify_external_catalogs: true
I20251024 19:48:01.974855 2020 raft_consensus.cc:515] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.979081 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe97e31310ae4733ba93de1b62a37842" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:01.979553 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet b9b1e0338fd14634a1e921735ae61289 from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:01.980278 1444 leader_election.cc:343] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:01.983238 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 1edc55caab904722bf19a1c71df601cf
I20251024 19:48:01.987439 2020 leader_election.cc:290] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:01.988286 1481 tablet_service.cc:1552] Processing DeleteTablet for tablet 1edc55caab904722bf19a1c71df601cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:01.988685 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a9c5f60f75d747b29b066b3e563b02a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:01.989331 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9c5f60f75d747b29b066b3e563b02a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:01.997771 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 1edc55caab904722bf19a1c71df601cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:01.998138 1767 ts_tablet_manager.cc:1403] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.151s user 0.004s sys 0.000s
I20251024 19:48:01.999045 1767 raft_consensus.cc:359] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:01.999325 1767 raft_consensus.cc:385] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:01.999480 1767 raft_consensus.cc:740] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:01.999770 1767 consensus_queue.cc:260] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.000449 1767 ts_tablet_manager.cc:1434] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:02.008885 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet e3b4e85f741849b2a6a3e8574396ff88
I20251024 19:48:02.009030 2098 raft_consensus.cc:2749] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:02.009418 1190 catalog_manager.cc:6719] ReplaceTablet: tablet e3b4e85f741849b2a6a3e8574396ff88 deleted and replaced by tablet 769f576810ab49368eb0fc7837b7da58
I20251024 19:48:02.010533 1775 tablet_replica.cc:333] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:02.010650 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet e3b4e85f741849b2a6a3e8574396ff88 with delete_type TABLET_DATA_DELETED (replaced by tablet 769f576810ab49368eb0fc7837b7da58) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.011106 1775 raft_consensus.cc:2243] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:02.011305 1775 raft_consensus.cc:2272] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:02.012156 1767 tablet_bootstrap.cc:492] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:02.015767 1775 ts_tablet_manager.cc:1916] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:02.016945 1478 tablet_service.cc:1552] Processing DeleteTablet for tablet e3b4e85f741849b2a6a3e8574396ff88 with delete_type TABLET_DATA_DELETED (replaced by tablet 769f576810ab49368eb0fc7837b7da58) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.020205 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2aa9405152c14d538c7164b50b336c7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.021129 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2aa9405152c14d538c7164b50b336c7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.022481 1766 tablet_bootstrap.cc:492] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:02.022706 1766 ts_tablet_manager.cc:1403] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.151s user 0.006s sys 0.003s
I20251024 19:48:02.022629 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_690da5d1c64e4889b9620a19138f55cc" } new_table_name: "default.Table_dedd7e29ac324761b962fbd47c7c25e9" modify_external_catalogs: true
I20251024 19:48:02.023643 1766 raft_consensus.cc:359] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.023926 1766 raft_consensus.cc:385] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:02.024235 1766 raft_consensus.cc:740] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:02.024502 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_65097482f924409b87d7f4dcb5029849"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:02.024608 1766 consensus_queue.cc:260] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:02.025014 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65097482f924409b87d7f4dcb5029849 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.012336 1776 log.cc:1199] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/f91316c030dc4f7da27977a9dde01f4b
I20251024 19:48:02.026289 1776 ts_tablet_manager.cc:1950] T f91316c030dc4f7da27977a9dde01f4b P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:02.017349 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet c292ce055def4964ab143252c6b2b87e with delete_type TABLET_DATA_DELETED (replaced by tablet d9bf08ccf7054fedb9efcbb00354c71a) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.032039 1766 ts_tablet_manager.cc:1434] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251024 19:48:01.902311 1774 rpcz_store.cc:276] 1024 19:47:58.925122 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.925192 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:01.750454 (+2825262us) ts_tablet_manager.cc:1151] Deleting tablet c292ce055def4964ab143252c6b2b87e
1024 19:48:01.750470 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:01.848390 (+ 97920us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:01.870714 (+ 22324us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:01.883942 (+ 13228us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2824993}
I20251024 19:48:02.038151 1478 tablet_service.cc:1552] Processing DeleteTablet for tablet c292ce055def4964ab143252c6b2b87e with delete_type TABLET_DATA_DELETED (replaced by tablet d9bf08ccf7054fedb9efcbb00354c71a) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.041642 1766 tablet_bootstrap.cc:492] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:02.044085 1766 tablet_bootstrap.cc:654] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:02.044066 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe97e31310ae4733ba93de1b62a37842" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:02.044838 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet c292ce055def4964ab143252c6b2b87e (table default.Table_4a085504a0884b37b6787b3bcf7a847f [id=10833938cf88466b834698420d7c01e3]) successfully deleted
I20251024 19:48:02.045776 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_740632e4f743422b850972065ef1f415"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.046635 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_740632e4f743422b850972065ef1f415 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.055070 1776 tablet_replica.cc:333] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:02.055040 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet f91316c030dc4f7da27977a9dde01f4b (table default.Table_cdf2e8872b724c7aa86d5091eb52fb1c [id=a402387542654ccf8e0b04ca8ecebb1a]) successfully deleted
I20251024 19:48:02.055487 1371 raft_consensus.cc:3060] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.055488 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet e3b4e85f741849b2a6a3e8574396ff88 with delete_type TABLET_DATA_DELETED (replaced by tablet 769f576810ab49368eb0fc7837b7da58) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.057250 1768 tablet_bootstrap.cc:492] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:02.057395 1768 ts_tablet_manager.cc:1403] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.116s user 0.027s sys 0.031s
I20251024 19:48:02.057835 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3e6db58fac5e4af9b27bde6c223aa845
I20251024 19:48:02.058094 1768 raft_consensus.cc:359] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.058225 1768 raft_consensus.cc:385] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:02.058293 1768 raft_consensus.cc:740] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:02.058516 1768 consensus_queue.cc:260] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.058981 1768 ts_tablet_manager.cc:1434] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251024 19:48:02.059305 1768 tablet_bootstrap.cc:492] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:02.059458 1371 raft_consensus.cc:2468] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:02.061216 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet 1edc55caab904722bf19a1c71df601cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.062635 1478 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e6db58fac5e4af9b27bde6c223aa845 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.063135 1447 leader_election.cc:304] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:02.062641 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e6db58fac5e4af9b27bde6c223aa845 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.055285 1776 raft_consensus.cc:2243] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:02.063808 2020 raft_consensus.cc:2804] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:02.063854 1776 raft_consensus.cc:2272] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:02.064023 2020 raft_consensus.cc:697] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:02.064481 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 182) took 3107 ms. Trace:
I20251024 19:48:02.064589 1775 ts_tablet_manager.cc:1929] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:02.064394 2020 consensus_queue.cc:237] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.064741 1775 log.cc:1199] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/8721ee81e4e24153a0aad8ce99f872e1
I20251024 19:48:02.065097 1775 ts_tablet_manager.cc:1950] T 8721ee81e4e24153a0aad8ce99f872e1 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:02.065657 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e6db58fac5e4af9b27bde6c223aa845 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.066707 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 8721ee81e4e24153a0aad8ce99f872e1 (table default.Table_aa7c283553a841e9ab2fa21fb6485a11 [id=7ec9391c52b1424eb52850f2c6ba4779]) successfully deleted
I20251024 19:48:02.068161 1775 tablet_replica.cc:333] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:02.068490 1775 raft_consensus.cc:2243] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:02.064428 1776 ts_tablet_manager.cc:1916] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:02.068766 1775 raft_consensus.cc:2272] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:02.069149 2059 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc8f71a5b70749a4a3e3a599bcb3d22c. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:48:02.069367 2059 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 19:48:02.069409 1775 ts_tablet_manager.cc:1916] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:02.069535 2108 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc8f71a5b70749a4a3e3a599bcb3d22c)
I20251024 19:48:02.069808 2108 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:48:02.071749 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_19fdf66f02124c62b4dad9bdeb7de7c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.072496 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19fdf66f02124c62b4dad9bdeb7de7c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.063429 1767 tablet_bootstrap.cc:654] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:02.074621 2108 leader_election.cc:290] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82e82847f8904b59b03f446b7fce6233 (127.0.252.253:41375), dc8f71a5b70749a4a3e3a599bcb3d22c (127.0.252.252:36627)
I20251024 19:48:02.086109 1185 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 6ca8f818440f4ae7a63c0901fcdcdf59
I20251024 19:48:02.086170 1775 ts_tablet_manager.cc:1929] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:02.086386 1775 log.cc:1199] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/15c08dcf379f4df9a0e461d6dcfeeac4
I20251024 19:48:02.086946 1775 ts_tablet_manager.cc:1950] T 15c08dcf379f4df9a0e461d6dcfeeac4 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:02.087718 1775 tablet_replica.cc:333] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:02.088025 1775 raft_consensus.cc:2243] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:02.088272 1775 raft_consensus.cc:2272] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:02.089058 1775 ts_tablet_manager.cc:1916] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:02.094283 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_09c1a7fb5d6e47da800d84d81b233bce" } new_table_name: "default.Table_e74966425ce8408da939910bc645012d" modify_external_catalogs: true
I20251024 19:48:02.102092 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2496f103436047dea5dabb577651249f
I20251024 19:48:02.103600 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 2496f103436047dea5dabb577651249f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.107245 1775 ts_tablet_manager.cc:1929] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:02.113759 1766 tablet_bootstrap.cc:492] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:02.113968 1766 ts_tablet_manager.cc:1403] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.000s
I20251024 19:48:02.115017 1766 raft_consensus.cc:359] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.115232 1766 raft_consensus.cc:385] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:02.115338 1766 raft_consensus.cc:740] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:02.115705 1766 consensus_queue.cc:260] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.116083 1768 tablet_bootstrap.cc:654] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:02.116829 1774 rpcz_store.cc:276] 1024 19:47:58.926508 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.927310 (+ 802us) service_pool.cc:225] Handling call
1024 19:48:02.033646 (+3106336us) ts_tablet_manager.cc:1151] Deleting tablet 030e690ce4df4818a6450ae293b0ffbd
1024 19:48:02.033656 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.033796 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3092321}
I20251024 19:48:02.117378 1766 ts_tablet_manager.cc:1434] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:02.118191 1774 tablet_replica.cc:333] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:02.118409 1774 raft_consensus.cc:2243] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:02.118664 1766 tablet_bootstrap.cc:492] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:02.119529 1774 pending_rounds.cc:70] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:02.119612 1774 pending_rounds.cc:77] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429303602298880 op_type: NO_OP noop_request { }
I20251024 19:48:02.119756 2098 raft_consensus.cc:493] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.120613 1766 tablet_bootstrap.cc:654] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:02.120608 1774 raft_consensus.cc:2889] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:02.120747 1774 raft_consensus.cc:2272] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:02.121651 2062 raft_consensus.cc:493] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.121788 2062 raft_consensus.cc:515] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.122476 2062 leader_election.cc:290] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:02.123121 1774 ts_tablet_manager.cc:1916] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:02.123672 1444 leader_election.cc:343] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: f91316c030dc4f7da27977a9dde01f4b
W20251024 19:48:02.124015 1447 leader_election.cc:343] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: f91316c030dc4f7da27977a9dde01f4b
I20251024 19:48:02.124145 1447 leader_election.cc:304] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:02.124641 2062 raft_consensus.cc:2749] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:02.125098 1775 log.cc:1199] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/95d7fbfa8ae54da49dd5603294faa7d3
I20251024 19:48:02.125326 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 96a6e0a4882c42399fd07f06ac21aa3f
I20251024 19:48:02.125656 1775 ts_tablet_manager.cc:1950] T 95d7fbfa8ae54da49dd5603294faa7d3 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:02.126744 1775 tablet_replica.cc:333] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:02.127168 1775 raft_consensus.cc:2243] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:02.127645 1775 raft_consensus.cc:2272] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:02.127646 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 96a6e0a4882c42399fd07f06ac21aa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.128301 2062 raft_consensus.cc:493] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.128477 2062 raft_consensus.cc:515] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.129181 2062 leader_election.cc:290] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:02.130031 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "369cebfda6ee43638b79f8652fa55855" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:02.130548 1648 raft_consensus.cc:2468] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:02.131069 1444 leader_election.cc:304] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:02.131225 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "369cebfda6ee43638b79f8652fa55855" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:02.131414 2062 raft_consensus.cc:2804] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:02.131646 2062 raft_consensus.cc:493] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:02.131753 2062 raft_consensus.cc:3060] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.135318 2062 raft_consensus.cc:515] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.137072 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "369cebfda6ee43638b79f8652fa55855" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:02.137908 2062 leader_election.cc:290] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:02.138243 2062 raft_consensus.cc:493] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.138592 2062 raft_consensus.cc:515] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.141635 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "369cebfda6ee43638b79f8652fa55855" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:02.141868 1648 raft_consensus.cc:3060] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.142071 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f05ad092936b4bafa3ae795904c2df78" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:02.142962 1444 leader_election.cc:343] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:02.144007 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f05ad092936b4bafa3ae795904c2df78" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:02.144583 1447 leader_election.cc:343] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:02.144691 1648 raft_consensus.cc:2468] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:02.144826 1447 leader_election.cc:304] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:02.145323 1444 leader_election.cc:304] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:02.149737 2062 leader_election.cc:290] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:02.150652 2062 raft_consensus.cc:2749] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:02.151224 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet b9b1e0338fd14634a1e921735ae61289 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.151880 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 2496f103436047dea5dabb577651249f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.150959 2020 raft_consensus.cc:2804] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:02.151968 1478 tablet_service.cc:1552] Processing DeleteTablet for tablet 2496f103436047dea5dabb577651249f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.152815 2020 raft_consensus.cc:697] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:02.152851 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 96a6e0a4882c42399fd07f06ac21aa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.153370 1478 tablet_service.cc:1552] Processing DeleteTablet for tablet 96a6e0a4882c42399fd07f06ac21aa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.153282 2020 consensus_queue.cc:237] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:02.154589 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 030e690ce4df4818a6450ae293b0ffbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 030e690ce4df4818a6450ae293b0ffbd already in progress: creating tablet
I20251024 19:48:02.154920 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 15c08dcf379f4df9a0e461d6dcfeeac4 (table default.Table_96c033f8e7ab4c919b38c80fabe3e367 [id=473fc6762db94b959bb9a260b11140f5]) successfully deleted
I20251024 19:48:02.156927 1767 tablet_bootstrap.cc:492] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:02.157290 1767 ts_tablet_manager.cc:1403] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.145s user 0.004s sys 0.000s
I20251024 19:48:02.157332 2098 raft_consensus.cc:515] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.158178 2098 leader_election.cc:290] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:02.158684 1767 raft_consensus.cc:359] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.159003 1767 raft_consensus.cc:385] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:02.159162 1767 raft_consensus.cc:740] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:02.159512 1767 consensus_queue.cc:260] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.161710 1775 ts_tablet_manager.cc:1916] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:02.162730 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "369cebfda6ee43638b79f8652fa55855" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:02.163021 1648 raft_consensus.cc:2393] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
I20251024 19:48:02.164270 1767 ts_tablet_manager.cc:1434] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251024 19:48:02.164621 1767 tablet_bootstrap.cc:492] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:02.164942 1371 raft_consensus.cc:2468] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:02.168421 1960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52566 (request call id 0) took 1665 ms. Trace:
I20251024 19:48:02.160424 2133 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251024 19:48:02.175282 1766 tablet_bootstrap.cc:492] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:02.177805 1766 ts_tablet_manager.cc:1403] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.004s
I20251024 19:48:02.178686 1766 raft_consensus.cc:359] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.179934 1766 raft_consensus.cc:385] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:02.180341 1766 raft_consensus.cc:740] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:02.180961 1766 consensus_queue.cc:260] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.183331 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_59612aa90b7540f6bf9cfc72e1f68fc1" } new_table_name: "default.Table_50e12cf5f39d4fa2983835ef0dbbc3d6" modify_external_catalogs: true
I20251024 19:48:02.187351 1766 ts_tablet_manager.cc:1434] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251024 19:48:02.187687 1766 tablet_bootstrap.cc:492] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:02.187919 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7d5114465a524d37b479ed5a6cebb938"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:02.189625 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet fe97e31310ae4733ba93de1b62a37842 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e6db58fac5e4af9b27bde6c223aa845) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.190739 1766 tablet_bootstrap.cc:654] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:02.192914 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03cdbd1b62b741249ad78562a7edfd22 consensus state has no leader
W20251024 19:48:02.195081 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d5114465a524d37b479ed5a6cebb938 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.203300 1775 ts_tablet_manager.cc:1929] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:02.205269 1775 log.cc:1199] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/110907ae943f45b5bcaa9b66ccdb84fd
I20251024 19:48:02.206010 1775 ts_tablet_manager.cc:1950] T 110907ae943f45b5bcaa9b66ccdb84fd P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:02.207005 1766 tablet_bootstrap.cc:492] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:02.207185 1766 ts_tablet_manager.cc:1403] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20251024 19:48:02.208038 1766 raft_consensus.cc:359] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.208249 1766 raft_consensus.cc:385] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:02.208338 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 110907ae943f45b5bcaa9b66ccdb84fd (table default.Table_2cbb0aad326e4bae85d4c5cb833150c6 [id=1acdc5d0bfb94e0faf47055b20d803b0]) successfully deleted
I20251024 19:48:02.208369 1766 raft_consensus.cc:740] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:02.208626 1766 consensus_queue.cc:260] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.209268 1766 ts_tablet_manager.cc:1434] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:02.209549 1766 tablet_bootstrap.cc:492] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:02.209560 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 5b96fc2309274048bed7a35e4bf4bae2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b96fc2309274048bed7a35e4bf4bae2 already in progress: creating tablet
W20251024 19:48:02.213820 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 16bc737104ab46b3b071a0760dbc847d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16bc737104ab46b3b071a0760dbc847d already in progress: creating tablet
W20251024 19:48:02.214805 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 59c7dbe9ff2c4869994c4f43d5676c30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59c7dbe9ff2c4869994c4f43d5676c30 already in progress: creating tablet
W20251024 19:48:02.215799 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 187bc968c2b049028dd25dfb0264c0aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 187bc968c2b049028dd25dfb0264c0aa already in progress: creating tablet
W20251024 19:48:02.217228 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce8c179e98944df0b510a0e8daf3920f consensus state has no leader
I20251024 19:48:02.217406 1768 tablet_bootstrap.cc:492] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:02.217628 1768 ts_tablet_manager.cc:1403] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.158s user 0.000s sys 0.009s
I20251024 19:48:02.219218 1768 raft_consensus.cc:359] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.219539 1768 raft_consensus.cc:385] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:02.219666 1768 raft_consensus.cc:740] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:02.219997 1768 consensus_queue.cc:260] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.220774 2048 raft_consensus.cc:493] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.220942 2048 raft_consensus.cc:515] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.221640 2048 leader_election.cc:290] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:02.221841 2048 raft_consensus.cc:493] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.221971 2048 raft_consensus.cc:515] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.222604 2048 leader_election.cc:290] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:02.223498 1768 ts_tablet_manager.cc:1434] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20251024 19:48:02.223941 1768 tablet_bootstrap.cc:492] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:02.226023 1768 tablet_bootstrap.cc:654] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:02.227229 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 639836c656f44a819036f4d00e055bd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 639836c656f44a819036f4d00e055bd9 already in progress: creating tablet
W20251024 19:48:02.228003 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9489a01434a24d9ca94791252950725c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9489a01434a24d9ca94791252950725c already in progress: creating tablet
I20251024 19:48:02.231662 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_405cbdb6a024494a81e147548e24d436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:02.234102 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet f05ad092936b4bafa3ae795904c2df78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.234797 1478 tablet_service.cc:1552] Processing DeleteTablet for tablet 369cebfda6ee43638b79f8652fa55855 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.235154 1775 tablet_replica.cc:333] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:02.235829 1478 tablet_service.cc:1552] Processing DeleteTablet for tablet b9b1e0338fd14634a1e921735ae61289 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:02.236162 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_405cbdb6a024494a81e147548e24d436 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.236426 1775 raft_consensus.cc:2243] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:02.236670 1775 raft_consensus.cc:2272] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:02.237401 1775 ts_tablet_manager.cc:1916] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:02.233865 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a4b63032d2224a6d95765cb93137f4a2" } modify_external_catalogs: true
I20251024 19:48:02.237800 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a4b63032d2224a6d95765cb93137f4a2" } modify_external_catalogs: true
I20251024 19:48:02.234186 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9ea4b3e0bf8d41ce844677ff6dc9ddfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.238629 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ea4b3e0bf8d41ce844677ff6dc9ddfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.238823 1774 ts_tablet_manager.cc:1929] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:02.238984 1774 log.cc:1199] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/97c6dce647064344b0b53a7ffaeafb75
I20251024 19:48:02.239423 1774 ts_tablet_manager.cc:1950] T 97c6dce647064344b0b53a7ffaeafb75 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:02.234553 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4c3e151bdb364b37ac92d4c3b767dd35" } new_table_name: "default.Table_7bc8cd4a930b44dfafa473297875594d" modify_external_catalogs: true
I20251024 19:48:02.234894 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_325e96b0d32e4a3aab388a916c29ec26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.241252 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_325e96b0d32e4a3aab388a916c29ec26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:02.242115 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 96792.5us
I20251024 19:48:02.242606 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "369cebfda6ee43638b79f8652fa55855" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:02.243640 1510 raft_consensus.cc:3060] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.247920 1308 leader_election.cc:304] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:02.248842 2098 raft_consensus.cc:3060] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.252183 2098 raft_consensus.cc:2749] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:02.252533 2098 consensus_queue.cc:1048] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:02.255566 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8b8ee145388d4588b50306be38739650"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.256151 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b8ee145388d4588b50306be38739650 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.257653 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet 16bc737104ab46b3b071a0760dbc847d from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:02.258281 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet dc1b6f0a377945089c50012f74d38564 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:02.258733 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet 2496f103436047dea5dabb577651249f from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:02.259177 1478 tablet_service.cc:1552] Processing DeleteTablet for tablet 90405a0c13bd4c229b011c90232f751d with delete_type TABLET_DATA_DELETED (replaced by tablet 59c7dbe9ff2c4869994c4f43d5676c30) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.262249 1767 tablet_bootstrap.cc:654] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:02.263926 1766 tablet_bootstrap.cc:654] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:02.273882 1934 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "43bec2f03fc849099e254f6335e9d532" instance_seqno: 1761335273959853) as {username='slave'} at 127.0.252.193:56151; Asking this server to re-register.
I20251024 19:48:02.275113 1418 heartbeater.cc:461] Registering TS with master...
I20251024 19:48:02.281512 1934 ts_manager.cc:194] Registered new tserver with Master: 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:02.282670 1776 ts_tablet_manager.cc:1929] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:02.283290 1775 ts_tablet_manager.cc:1929] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:02.284844 1775 log.cc:1199] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/2e956ea914c545d79be6d632bfc4ec57
I20251024 19:48:02.286123 1775 ts_tablet_manager.cc:1950] T 2e956ea914c545d79be6d632bfc4ec57 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:02.288182 1934 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "10a9536207e343f188e0ef709081d04e" instance_seqno: 1761335274554669) as {username='slave'} at 127.0.252.194:54175; Asking this server to re-register.
I20251024 19:48:02.290020 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 2e956ea914c545d79be6d632bfc4ec57 (table default.Table_3819751f36bf419e9edd0f1ff339713a [id=350b37bf42194fcfbb660bffe83e47c4]) successfully deleted
I20251024 19:48:02.291285 1478 tablet_service.cc:1505] Processing CreateTablet for tablet 2e7b3288182744969e217d4f174fe4ea (DEFAULT_TABLE table=default.Table_c4882268578941ccaf2c28be3074fdd2 [id=0b00aa7d2eaa4b1a967952b9a484fd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.291993 1478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e7b3288182744969e217d4f174fe4ea. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:02.293720 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28005301f9de48e5826f46c79aeb5fe5 consensus state has no leader
I20251024 19:48:02.294121 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet f3d80885fcc642b8a75da30cd4abecbd from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:02.240319 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 190) took 3304 ms. Trace:
I20251024 19:48:02.298575 1774 rpcz_store.cc:276] 1024 19:47:58.936160 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.936345 (+ 185us) service_pool.cc:225] Handling call
1024 19:48:02.117144 (+3180799us) ts_tablet_manager.cc:1151] Deleting tablet 97c6dce647064344b0b53a7ffaeafb75
1024 19:48:02.117158 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.216578 (+ 99420us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:02.238693 (+ 22115us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:02.240257 (+ 1564us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3116928}
I20251024 19:48:02.299118 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 191) took 3303 ms. Trace:
I20251024 19:48:02.299118 2141 raft_consensus.cc:493] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.299221 1774 rpcz_store.cc:276] 1024 19:47:58.995563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:58.996489 (+ 926us) service_pool.cc:225] Handling call
1024 19:48:02.298928 (+3302439us) ts_tablet_manager.cc:1151] Deleting tablet a48249f1824a40b087b80a563816d485
1024 19:48:02.298941 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.299061 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3209681}
I20251024 19:48:02.299283 2141 raft_consensus.cc:515] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.299616 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 201) took 2846 ms. Trace:
I20251024 19:48:02.299707 1774 rpcz_store.cc:276] 1024 19:47:59.453554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.453626 (+ 72us) service_pool.cc:225] Handling call
1024 19:48:02.299475 (+2845849us) ts_tablet_manager.cc:1151] Deleting tablet ca0527af718f494a8bbcc66e7b325bed
1024 19:48:02.299485 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.299567 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2845561}
I20251024 19:48:02.300222 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 202) took 2846 ms. Trace:
I20251024 19:48:02.300318 1774 rpcz_store.cc:276] 1024 19:47:59.453713 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.453775 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:02.299943 (+2846168us) ts_tablet_manager.cc:1151] Deleting tablet 9fffc42ddda14bd28800917ff2f0ced5
1024 19:48:02.300082 (+ 139us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.300174 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2845993}
I20251024 19:48:02.300789 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 203) took 2846 ms. Trace:
I20251024 19:48:02.300882 1774 rpcz_store.cc:276] 1024 19:47:59.453856 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.453976 (+ 120us) service_pool.cc:225] Handling call
1024 19:48:02.300641 (+2846665us) ts_tablet_manager.cc:1151] Deleting tablet 7c357a2a81064b3891afda6d252898b9
1024 19:48:02.300649 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.300743 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2846501}
I20251024 19:48:02.301359 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 197) took 2859 ms. Trace:
I20251024 19:48:02.301453 1774 rpcz_store.cc:276] 1024 19:47:59.441911 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.441995 (+ 84us) service_pool.cc:225] Handling call
1024 19:48:02.301188 (+2859193us) ts_tablet_manager.cc:1151] Deleting tablet 5fd7726c532145c6b44ea9a37544510d
1024 19:48:02.301196 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.301304 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2846990}
I20251024 19:48:02.302785 2141 leader_election.cc:290] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:02.303249 1447 leader_election.cc:343] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 90405a0c13bd4c229b011c90232f751d
I20251024 19:48:02.305039 1478 tablet_service.cc:1505] Processing CreateTablet for tablet 45cff9bcc4924e0081518c284f074940 (DEFAULT_TABLE table=default.Table_34ef66bee6bd460b92cf30b469b3f179 [id=32d4cdae3aac408a88b0a207c8c08fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.305439 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90405a0c13bd4c229b011c90232f751d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:02.305608 1478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45cff9bcc4924e0081518c284f074940. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.305763 1648 raft_consensus.cc:2468] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:02.306178 1776 log.cc:1199] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/9820e68ad5264df1a79b22f3b5369c5a
I20251024 19:48:02.306183 1444 leader_election.cc:304] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:02.306699 1776 ts_tablet_manager.cc:1950] T 9820e68ad5264df1a79b22f3b5369c5a P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:02.306921 2141 raft_consensus.cc:2804] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:02.307098 2141 raft_consensus.cc:493] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:02.307250 2141 raft_consensus.cc:3060] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.308666 1776 tablet_replica.cc:333] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:02.308897 1776 raft_consensus.cc:2243] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:02.309100 1776 raft_consensus.cc:2272] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:02.310065 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 45cff9bcc4924e0081518c284f074940 (DEFAULT_TABLE table=default.Table_34ef66bee6bd460b92cf30b469b3f179 [id=32d4cdae3aac408a88b0a207c8c08fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.310747 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45cff9bcc4924e0081518c284f074940. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.311064 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe97e31310ae4733ba93de1b62a37842" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:02.311478 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c292ce055def4964ab143252c6b2b87e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:02.311687 1511 raft_consensus.cc:2468] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:02.312696 1586 leader_election.cc:304] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:02.312963 2048 raft_consensus.cc:2804] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:02.313081 2048 raft_consensus.cc:493] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:02.313066 1486 tablet_service.cc:1505] Processing CreateTablet for tablet b632acc1348e49f8a1eea019c007780c (DEFAULT_TABLE table=default.Table_82dcb5dcc93042b4a560b867c853f30d [id=dc00ee30038e49fb9f86499cfb985a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.313210 2048 raft_consensus.cc:3060] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.313505 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b632acc1348e49f8a1eea019c007780c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:02.316278 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b20a6845ae1645c0b51f096d471d4ccc consensus state has no leader
I20251024 19:48:02.317440 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c777799df90e491992909e8a45205a06" } modify_external_catalogs: true
I20251024 19:48:02.317648 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c777799df90e491992909e8a45205a06" } modify_external_catalogs: true
I20251024 19:48:02.319159 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 2e7b3288182744969e217d4f174fe4ea (DEFAULT_TABLE table=default.Table_c4882268578941ccaf2c28be3074fdd2 [id=0b00aa7d2eaa4b1a967952b9a484fd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.319566 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e7b3288182744969e217d4f174fe4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.304214 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 2e7b3288182744969e217d4f174fe4ea (DEFAULT_TABLE table=default.Table_c4882268578941ccaf2c28be3074fdd2 [id=0b00aa7d2eaa4b1a967952b9a484fd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.306744 2130 raft_consensus.cc:493] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.309724 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 45cff9bcc4924e0081518c284f074940 (DEFAULT_TABLE table=default.Table_34ef66bee6bd460b92cf30b469b3f179 [id=32d4cdae3aac408a88b0a207c8c08fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.320827 2130 raft_consensus.cc:515] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.313766 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe97e31310ae4733ba93de1b62a37842" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:02.321791 1371 raft_consensus.cc:2393] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
I20251024 19:48:02.322639 2130 leader_election.cc:290] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:02.322775 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c04082bd1b444b468c177bef239701c6" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:02.323323 1308 leader_election.cc:343] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:02.323402 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 9820e68ad5264df1a79b22f3b5369c5a (table default.Table_0680f9139a8a436a80fce09c095912ce [id=071147c9baed46f8a30f4c2fbc0fbfaa]) successfully deleted
I20251024 19:48:02.323655 1557 heartbeater.cc:461] Registering TS with master...
W20251024 19:48:02.323730 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 59c7dbe9ff2c4869994c4f43d5676c30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59c7dbe9ff2c4869994c4f43d5676c30 already in progress: creating tablet
I20251024 19:48:02.323889 1617 tablet_service.cc:1505] Processing CreateTablet for tablet d1d26e02ad2945b39cf27738ed5b7cf4 (DEFAULT_TABLE table=default.Table_2c02fd9cab9b479d8876a6dcbd586b17 [id=3660d29be985453fa1741ee293ac2a14]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:02.323946 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 639836c656f44a819036f4d00e055bd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 639836c656f44a819036f4d00e055bd9 already in progress: creating tablet
W20251024 19:48:02.324175 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9489a01434a24d9ca94791252950725c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9489a01434a24d9ca94791252950725c already in progress: creating tablet
I20251024 19:48:02.324288 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1d26e02ad2945b39cf27738ed5b7cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.324230 2020 raft_consensus.cc:493] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251024 19:48:02.324417 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 187bc968c2b049028dd25dfb0264c0aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 187bc968c2b049028dd25dfb0264c0aa already in progress: creating tablet
I20251024 19:48:02.324376 2020 raft_consensus.cc:515] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.324713 1624 tablet_service.cc:1505] Processing CreateTablet for tablet b632acc1348e49f8a1eea019c007780c (DEFAULT_TABLE table=default.Table_82dcb5dcc93042b4a560b867c853f30d [id=dc00ee30038e49fb9f86499cfb985a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.324945 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 23d4dd0b42d343cb9354ac698a8c761f (DEFAULT_TABLE table=default.Table_628b2afa9401426980be0eae5dffd917 [id=5b96ec1514ae41e3a0103e4b381aab61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.325055 2020 leader_election.cc:290] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:02.325927 1586 leader_election.cc:343] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: c292ce055def4964ab143252c6b2b87e
W20251024 19:48:02.326073 1447 leader_election.cc:343] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 9820e68ad5264df1a79b22f3b5369c5a
I20251024 19:48:02.327334 1774 tablet_replica.cc:333] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:02.327673 1774 raft_consensus.cc:2243] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:02.328006 1774 raft_consensus.cc:2272] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:02.328860 1131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef8906492b1a4368936fac95c11e1f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 335 } } ignore_live_leader: false dest_uuid: "82e82847f8904b59b03f446b7fce6233" is_pre_election: true
I20251024 19:48:02.329116 1774 ts_tablet_manager.cc:1916] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:02.329457 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45cff9bcc4924e0081518c284f074940. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.329520 1766 tablet_bootstrap.cc:492] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:02.329715 1766 ts_tablet_manager.cc:1403] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.120s user 0.005s sys 0.003s
I20251024 19:48:02.330556 1766 raft_consensus.cc:359] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:02.330755 1444 leader_election.cc:343] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 9820e68ad5264df1a79b22f3b5369c5a
I20251024 19:48:02.330868 1444 leader_election.cc:304] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:02.331249 2020 raft_consensus.cc:2749] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:02.331230 1766 raft_consensus.cc:385] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:02.331490 1766 raft_consensus.cc:740] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:02.331738 1766 consensus_queue.cc:260] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:02.332311 1766 ts_tablet_manager.cc:1434] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:02.332698 1766 tablet_bootstrap.cc:492] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:02.333822 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e7b3288182744969e217d4f174fe4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.335455 1630 tablet_service.cc:1505] Processing CreateTablet for tablet da7d38b63708427982591da1014aca81 (DEFAULT_TABLE table=default.Table_c7ecd114a8d44f5da18755c9a5ebed07 [id=334320a439bf4611ba5b1aa660844023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.335598 1766 tablet_bootstrap.cc:654] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:02.335803 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b (DEFAULT_TABLE table=default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.342002 1614 tablet_service.cc:1505] Processing CreateTablet for tablet ee5356b5503c4425bf6648ade5f6330e (DEFAULT_TABLE table=default.Table_40abb33fa043417bb208476294761da9 [id=239fc223508c45a980b6254769bc8c12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.342386 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d4dd0b42d343cb9354ac698a8c761f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.343539 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 4100abc9ff12471e9a4959b6b24f7e99 (DEFAULT_TABLE table=default.Table_81453b8401974cbcaa50e97a3c3bf12d [id=0e3c3f6ff360497c93fac944f351702c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.343930 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4100abc9ff12471e9a4959b6b24f7e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.344023 2048 raft_consensus.cc:515] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.344729 2048 leader_election.cc:290] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:02.345108 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee5356b5503c4425bf6648ade5f6330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.343843 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c04082bd1b444b468c177bef239701c6" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:02.346508 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.348063 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da7d38b63708427982591da1014aca81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.349112 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b632acc1348e49f8a1eea019c007780c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:02.350186 1305 leader_election.cc:343] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:02.350342 1305 leader_election.cc:304] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:02.350962 2098 raft_consensus.cc:2749] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:02.351110 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 23d4dd0b42d343cb9354ac698a8c761f (DEFAULT_TABLE table=default.Table_628b2afa9401426980be0eae5dffd917 [id=5b96ec1514ae41e3a0103e4b381aab61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.351382 1776 ts_tablet_manager.cc:1916] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:02.351720 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d4dd0b42d343cb9354ac698a8c761f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.353228 1347 tablet_service.cc:1505] Processing CreateTablet for tablet b632acc1348e49f8a1eea019c007780c (DEFAULT_TABLE table=default.Table_82dcb5dcc93042b4a560b867c853f30d [id=dc00ee30038e49fb9f86499cfb985a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.353628 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b632acc1348e49f8a1eea019c007780c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:02.357613 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a1fbbba88a294adfbf97c0343d86698e consensus state has no leader
I20251024 19:48:02.360394 2141 raft_consensus.cc:515] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.362248 2130 raft_consensus.cc:493] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.362485 2130 raft_consensus.cc:515] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:02.363523 2130 leader_election.cc:290] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:02.363984 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 23d4dd0b42d343cb9354ac698a8c761f (DEFAULT_TABLE table=default.Table_628b2afa9401426980be0eae5dffd917 [id=5b96ec1514ae41e3a0103e4b381aab61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.364483 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d4dd0b42d343cb9354ac698a8c761f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.367936 1586 leader_election.cc:304] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:02.377077 1341 tablet_service.cc:1505] Processing CreateTablet for tablet d1d26e02ad2945b39cf27738ed5b7cf4 (DEFAULT_TABLE table=default.Table_2c02fd9cab9b479d8876a6dcbd586b17 [id=3660d29be985453fa1741ee293ac2a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.377557 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1d26e02ad2945b39cf27738ed5b7cf4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:02.381986 1305 consensus_peers.cc:597] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 979dd87a8c66480ea314fcc75fa878c3. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:02.385442 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b (DEFAULT_TABLE table=default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.385885 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.388134 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 5d129bdc876e425aabcf1c99997e319a
I20251024 19:48:02.388127 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f3d80885fcc642b8a75da30cd4abecbd
I20251024 19:48:02.390224 1185 catalog_manager.cc:6719] ReplaceTablet: tablet f3d80885fcc642b8a75da30cd4abecbd deleted and replaced by tablet 780442b310e6409498e6bbe4256280a4
I20251024 19:48:02.393080 1350 tablet_service.cc:1505] Processing CreateTablet for tablet ee5356b5503c4425bf6648ade5f6330e (DEFAULT_TABLE table=default.Table_40abb33fa043417bb208476294761da9 [id=239fc223508c45a980b6254769bc8c12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.393790 1347 tablet_service.cc:1505] Processing CreateTablet for tablet da7d38b63708427982591da1014aca81 (DEFAULT_TABLE table=default.Table_c7ecd114a8d44f5da18755c9a5ebed07 [id=334320a439bf4611ba5b1aa660844023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.394217 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee5356b5503c4425bf6648ade5f6330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.395334 2146 raft_consensus.cc:3060] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.399649 2146 raft_consensus.cc:2749] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:02.400991 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 129debb55f4a4b7295a8ad88e55ebc83
I20251024 19:48:02.403863 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da7d38b63708427982591da1014aca81. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:02.406872 1586 leader_election.cc:343] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: c292ce055def4964ab143252c6b2b87e
I20251024 19:48:02.409114 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2346c03802a245a2af8c086b626d660d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:02.410387 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 4100abc9ff12471e9a4959b6b24f7e99 (DEFAULT_TABLE table=default.Table_81453b8401974cbcaa50e97a3c3bf12d [id=0e3c3f6ff360497c93fac944f351702c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.410804 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4100abc9ff12471e9a4959b6b24f7e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.414085 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 97c6dce647064344b0b53a7ffaeafb75 (table default.Table_85e6c4c107c94e7abfc7c3396aaf6e81 [id=1b078c2814dd4ff99c1e2ee01d0159cf]) successfully deleted
W20251024 19:48:02.414364 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet a48249f1824a40b087b80a563816d485 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a48249f1824a40b087b80a563816d485 already in progress: creating tablet
W20251024 19:48:02.414565 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet ca0527af718f494a8bbcc66e7b325bed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca0527af718f494a8bbcc66e7b325bed already in progress: creating tablet
W20251024 19:48:02.414786 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9fffc42ddda14bd28800917ff2f0ced5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fffc42ddda14bd28800917ff2f0ced5 already in progress: creating tablet
W20251024 19:48:02.414968 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7c357a2a81064b3891afda6d252898b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c357a2a81064b3891afda6d252898b9 already in progress: creating tablet
W20251024 19:48:02.415160 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 5fd7726c532145c6b44ea9a37544510d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fd7726c532145c6b44ea9a37544510d already in progress: creating tablet
I20251024 19:48:02.416790 1767 tablet_bootstrap.cc:492] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:02.361723 1487 tablet_service.cc:1505] Processing CreateTablet for tablet d1d26e02ad2945b39cf27738ed5b7cf4 (DEFAULT_TABLE table=default.Table_2c02fd9cab9b479d8876a6dcbd586b17 [id=3660d29be985453fa1741ee293ac2a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.421167 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1d26e02ad2945b39cf27738ed5b7cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.363538 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2346c03802a245a2af8c086b626d660d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:02.364470 1766 tablet_bootstrap.cc:492] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:02.425729 1511 raft_consensus.cc:2468] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:02.369030 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c292ce055def4964ab143252c6b2b87e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:02.425796 1766 ts_tablet_manager.cc:1403] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.002s
I20251024 19:48:02.425962 1508 raft_consensus.cc:3060] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.390503 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b (DEFAULT_TABLE table=default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.426524 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.426772 1766 raft_consensus.cc:359] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.426980 1766 raft_consensus.cc:385] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:02.427255 1766 raft_consensus.cc:740] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:02.427596 1766 consensus_queue.cc:260] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.391265 1490 tablet_service.cc:1505] Processing CreateTablet for tablet da7d38b63708427982591da1014aca81 (DEFAULT_TABLE table=default.Table_c7ecd114a8d44f5da18755c9a5ebed07 [id=334320a439bf4611ba5b1aa660844023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.428630 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da7d38b63708427982591da1014aca81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.428668 1766 ts_tablet_manager.cc:1434] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:02.429049 1766 tablet_bootstrap.cc:492] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:02.392350 1480 tablet_service.cc:1505] Processing CreateTablet for tablet ee5356b5503c4425bf6648ade5f6330e (DEFAULT_TABLE table=default.Table_40abb33fa043417bb208476294761da9 [id=239fc223508c45a980b6254769bc8c12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.430910 1308 leader_election.cc:304] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:02.410121 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 4100abc9ff12471e9a4959b6b24f7e99 (DEFAULT_TABLE table=default.Table_81453b8401974cbcaa50e97a3c3bf12d [id=0e3c3f6ff360497c93fac944f351702c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.432021 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d129bdc876e425aabcf1c99997e319a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.432787 2130 raft_consensus.cc:2804] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:02.433451 2130 raft_consensus.cc:493] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:02.434088 2130 raft_consensus.cc:3060] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.434892 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee5356b5503c4425bf6648ade5f6330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.437704 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90405a0c13bd4c229b011c90232f751d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:02.438601 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet f3d80885fcc642b8a75da30cd4abecbd with delete_type TABLET_DATA_DELETED (replaced by tablet 780442b310e6409498e6bbe4256280a4) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.439088 1621 tablet_service.cc:1505] Processing CreateTablet for tablet fc7ccf1495924782b9b6ac1173bc73a1 (DEFAULT_TABLE table=default.Table_19fdf66f02124c62b4dad9bdeb7de7c4 [id=991b0e462eba40ec882e422e60b43f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.439678 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc7ccf1495924782b9b6ac1173bc73a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.440626 1767 ts_tablet_manager.cc:1403] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.254s user 0.001s sys 0.005s
I20251024 19:48:02.441458 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e962c1d3850943a0a9e86f5882d1c518" } modify_external_catalogs: true
I20251024 19:48:02.441746 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e962c1d3850943a0a9e86f5882d1c518" } modify_external_catalogs: true
I20251024 19:48:02.441458 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4a085504a0884b37b6787b3bcf7a847f" } modify_external_catalogs: true
I20251024 19:48:02.443449 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4a085504a0884b37b6787b3bcf7a847f" } modify_external_catalogs: true
I20251024 19:48:02.441756 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2aa9405152c14d538c7164b50b336c7f" } modify_external_catalogs: true
I20251024 19:48:02.444938 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2aa9405152c14d538c7164b50b336c7f" } modify_external_catalogs: true
I20251024 19:48:02.446240 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_48177e84ca024d49a2fd4e205a08a2a9" } new_table_name: "default.Table_03760ade20c7464f81dd428a3eb734fb" modify_external_catalogs: true
I20251024 19:48:02.447510 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_13f216b61d134e4cba15849361a28394"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.448165 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13f216b61d134e4cba15849361a28394 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.449873 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_511ba1e13b8f43e299835a4c30063587"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:02.450083 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8b66ef90eef047bfbf665db21ee48274"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.450356 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_511ba1e13b8f43e299835a4c30063587 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:02.450596 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b66ef90eef047bfbf665db21ee48274 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.451859 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f46ed7e555b3432b95a3d9b829968b8e" } new_table_name: "default.Table_b09229e0e1254adeacfed952befaa65a" modify_external_catalogs: true
I20251024 19:48:02.453226 1202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef8906492b1a4368936fac95c11e1f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 335 } } ignore_live_leader: false dest_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" is_pre_election: true
I20251024 19:48:02.453791 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet 030e690ce4df4818a6450ae293b0ffbd from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:02.453855 1916 leader_election.cc:304] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef8906492b1a4368936fac95c11e1f73; no voters: 82e82847f8904b59b03f446b7fce6233, dc8f71a5b70749a4a3e3a599bcb3d22c
I20251024 19:48:02.454823 2108 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:48:02.459295 1767 raft_consensus.cc:359] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.459486 1767 raft_consensus.cc:385] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:02.459563 1767 raft_consensus.cc:740] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:02.459776 1767 consensus_queue.cc:260] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.456828 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 7be9290daabb404f955f05922d897fbf (DEFAULT_TABLE table=default.Table_65097482f924409b87d7f4dcb5029849 [id=867c6e52ff2f4eb3a45495738770709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.461382 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7be9290daabb404f955f05922d897fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.462033 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8631b5d780614ecdb746a74a4d97b8b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.462651 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8631b5d780614ecdb746a74a4d97b8b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.464648 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_03318e0d18d34db48da05226fd40029a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.465353 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03318e0d18d34db48da05226fd40029a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.468230 2130 raft_consensus.cc:515] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:02.433200 1305 leader_election.cc:343] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:02.444224 1774 ts_tablet_manager.cc:1929] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:02.469161 1774 log.cc:1199] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/9623d7baf0174a019e2353a5effef081
I20251024 19:48:02.469605 1774 ts_tablet_manager.cc:1950] T 9623d7baf0174a019e2353a5effef081 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:02.470016 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 198) took 3018 ms. Trace:
I20251024 19:48:02.470146 1774 rpcz_store.cc:276] 1024 19:47:59.451493 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.451564 (+ 71us) service_pool.cc:225] Handling call
1024 19:48:02.327162 (+2875598us) ts_tablet_manager.cc:1151] Deleting tablet 9623d7baf0174a019e2353a5effef081
1024 19:48:02.327203 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.374745 (+ 47542us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:02.444086 (+ 69341us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:02.469975 (+ 25889us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2871665}
I20251024 19:48:02.470642 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 200) took 3017 ms. Trace:
I20251024 19:48:02.470760 1774 rpcz_store.cc:276] 1024 19:47:59.453399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.453456 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:02.470453 (+3016997us) ts_tablet_manager.cc:1151] Deleting tablet 73ffe342f541447d8519ee24bc98a0f3
1024 19:48:02.470463 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.470603 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3013697}
I20251024 19:48:02.471115 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 199) took 3017 ms. Trace:
I20251024 19:48:02.471222 1774 rpcz_store.cc:276] 1024 19:47:59.453237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.454689 (+ 1452us) service_pool.cc:225] Handling call
1024 19:48:02.470980 (+3016291us) ts_tablet_manager.cc:1151] Deleting tablet b6661b9e9a174849b6093d09cd8c1251
1024 19:48:02.470989 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.471074 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3012892}
I20251024 19:48:02.471490 1774 tablet_replica.cc:333] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:02.471732 1774 raft_consensus.cc:2243] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:02.472020 1774 raft_consensus.cc:2272] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:02.472334 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 607ae453221d422e9d07882830c02275
I20251024 19:48:02.472671 1774 ts_tablet_manager.cc:1916] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:02.475986 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d129bdc876e425aabcf1c99997e319a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.476289 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet f3d80885fcc642b8a75da30cd4abecbd with delete_type TABLET_DATA_DELETED (replaced by tablet 780442b310e6409498e6bbe4256280a4) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.476890 1346 tablet_service.cc:1505] Processing CreateTablet for tablet fc7ccf1495924782b9b6ac1173bc73a1 (DEFAULT_TABLE table=default.Table_19fdf66f02124c62b4dad9bdeb7de7c4 [id=991b0e462eba40ec882e422e60b43f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.477384 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc7ccf1495924782b9b6ac1173bc73a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.481983 1350 tablet_service.cc:1505] Processing CreateTablet for tablet ee3c521876324808857f7724028c895f (DEFAULT_TABLE table=default.Table_740632e4f743422b850972065ef1f415 [id=f49bfd0c428d44f69a3d97a6c51a380a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.482373 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet d9bf08ccf7054fedb9efcbb00354c71a
I20251024 19:48:02.482482 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee3c521876324808857f7724028c895f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.485991 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 7be9290daabb404f955f05922d897fbf (DEFAULT_TABLE table=default.Table_65097482f924409b87d7f4dcb5029849 [id=867c6e52ff2f4eb3a45495738770709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.486444 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7be9290daabb404f955f05922d897fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.489188 1767 ts_tablet_manager.cc:1434] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20251024 19:48:02.489960 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 129debb55f4a4b7295a8ad88e55ebc83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.491086 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 9623d7baf0174a019e2353a5effef081 (table default.Table_7f14fdc529ec48d68d03ca041f64a53f [id=281060b712614bbeb46a006b6e34a91e]) successfully deleted
I20251024 19:48:02.491874 1934 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "af3952de050140d383f6b0de7666fa97" instance_seqno: 1761335275140404) as {username='slave'} at 127.0.252.195:44055; Asking this server to re-register.
I20251024 19:48:02.492918 1696 heartbeater.cc:461] Registering TS with master...
W20251024 19:48:02.494455 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 73ffe342f541447d8519ee24bc98a0f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73ffe342f541447d8519ee24bc98a0f3 already in progress: creating tablet
W20251024 19:48:02.495543 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b6661b9e9a174849b6093d09cd8c1251 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6661b9e9a174849b6093d09cd8c1251 already in progress: creating tablet
I20251024 19:48:02.496748 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 129debb55f4a4b7295a8ad88e55ebc83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.497587 1934 ts_manager.cc:194] Registered new tserver with Master: af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:02.497856 1178 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet fdfa5eea3b674a3389a156d1c72204df (table default.Table_73923b0648674b2891cbbf4b90d3f106 [id=96ea5a86f92b4d18b0d74f1092bd17b3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fdfa5eea3b674a3389a156d1c72204df consensus state has no leader
I20251024 19:48:02.501044 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 607ae453221d422e9d07882830c02275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.503331 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d129bdc876e425aabcf1c99997e319a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.504436 2141 leader_election.cc:290] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:02.505335 1444 leader_election.cc:343] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: Tablet not RUNNING: STOPPED
W20251024 19:48:02.505609 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 67143.6us
I20251024 19:48:02.505847 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bf08ccf7054fedb9efcbb00354c71a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.504302 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2346c03802a245a2af8c086b626d660d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:02.507896 1481 tablet_service.cc:1552] Processing DeleteTablet for tablet 129debb55f4a4b7295a8ad88e55ebc83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.508801 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet f3d80885fcc642b8a75da30cd4abecbd with delete_type TABLET_DATA_DELETED (replaced by tablet 780442b310e6409498e6bbe4256280a4) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:02.509215 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2346c03802a245a2af8c086b626d660d consensus state has no leader
I20251024 19:48:02.509384 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2346c03802a245a2af8c086b626d660d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:02.509689 1511 raft_consensus.cc:3060] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.510973 1481 tablet_service.cc:1505] Processing CreateTablet for tablet ee3c521876324808857f7724028c895f (DEFAULT_TABLE table=default.Table_740632e4f743422b850972065ef1f415 [id=f49bfd0c428d44f69a3d97a6c51a380a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.511401 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet f05ad092936b4bafa3ae795904c2df78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.511562 1486 tablet_service.cc:1505] Processing CreateTablet for tablet fc7ccf1495924782b9b6ac1173bc73a1 (DEFAULT_TABLE table=default.Table_19fdf66f02124c62b4dad9bdeb7de7c4 [id=991b0e462eba40ec882e422e60b43f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.512748 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 7be9290daabb404f955f05922d897fbf (DEFAULT_TABLE table=default.Table_65097482f924409b87d7f4dcb5029849 [id=867c6e52ff2f4eb3a45495738770709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.516391 1767 tablet_bootstrap.cc:492] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:02.517331 1508 raft_consensus.cc:2468] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:02.517455 1630 tablet_service.cc:1505] Processing CreateTablet for tablet ee3c521876324808857f7724028c895f (DEFAULT_TABLE table=default.Table_740632e4f743422b850972065ef1f415 [id=f49bfd0c428d44f69a3d97a6c51a380a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.517870 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 607ae453221d422e9d07882830c02275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.520164 1766 tablet_bootstrap.cc:654] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:02.521395 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bf08ccf7054fedb9efcbb00354c71a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.525187 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 80c58e2439dc490c824b5f0023b5434b
I20251024 19:48:02.525386 1511 raft_consensus.cc:2468] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
W20251024 19:48:02.525627 1447 leader_election.cc:343] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 90405a0c13bd4c229b011c90232f751d
I20251024 19:48:02.525856 1447 leader_election.cc:304] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:02.527107 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 80c58e2439dc490c824b5f0023b5434b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.527463 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 80c58e2439dc490c824b5f0023b5434b (DEFAULT_TABLE table=default.Table_2aa9405152c14d538c7164b50b336c7f [id=c00489e309ba4ada921f101a95bc8b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.530118 1491 tablet_service.cc:1505] Processing CreateTablet for tablet e38ef2ac19d14f9c91fa8abe86cc7987 (DEFAULT_TABLE table=default.Table_ea7e190995344cb8a1c35c95fc35b820 [id=50a4f64cc7504ea190832dd265c81caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.530467 2141 raft_consensus.cc:2749] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20251024 19:48:02.532660 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d692e1988d13460fb1d54e930e17fc9d consensus state has no leader
I20251024 19:48:02.535430 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 58a8f4f90f794e83a158e0026e5cea7f (DEFAULT_TABLE table=default.Table_8255c10f050f4670a31e7c175a36cb62 [id=3fe81a9c4a134e908f4f2599c4b0cbf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.539108 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 57f522a825264ea7b215304dbb7aa2d5 (DEFAULT_TABLE table=default.Table_6a4dbd8e2b264bbca433946dfb1890b7 [id=9fec2fa31d474c5f8bc3b39041ed30bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.540455 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 80c58e2439dc490c824b5f0023b5434b (DEFAULT_TABLE table=default.Table_2aa9405152c14d538c7164b50b336c7f [id=c00489e309ba4ada921f101a95bc8b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.541829 1586 leader_election.cc:304] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:02.542816 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 769f576810ab49368eb0fc7837b7da58 (DEFAULT_TABLE table=default.Table_b905a1285caa407e9fb08e81639c9bc8 [id=1da47e6010f548d1b3d8c538150a8a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.542899 2146 raft_consensus.cc:2804] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:02.543044 2146 raft_consensus.cc:697] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:02.544340 2146 consensus_queue.cc:237] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.545698 1768 tablet_bootstrap.cc:492] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:02.545715 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 80c58e2439dc490c824b5f0023b5434b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.545888 1768 ts_tablet_manager.cc:1403] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.322s user 0.004s sys 0.009s
I20251024 19:48:02.546667 2130 leader_election.cc:290] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:02.547281 1308 leader_election.cc:304] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:02.547736 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 769f576810ab49368eb0fc7837b7da58 (DEFAULT_TABLE table=default.Table_b905a1285caa407e9fb08e81639c9bc8 [id=1da47e6010f548d1b3d8c538150a8a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.548568 1775 tablet_replica.cc:333] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:02.548902 1775 raft_consensus.cc:2243] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:02.549353 1775 raft_consensus.cc:2272] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:02.549475 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee3c521876324808857f7724028c895f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.550305 1775 ts_tablet_manager.cc:1916] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:02.551103 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 769f576810ab49368eb0fc7837b7da58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.552073 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 58a8f4f90f794e83a158e0026e5cea7f (DEFAULT_TABLE table=default.Table_8255c10f050f4670a31e7c175a36cb62 [id=3fe81a9c4a134e908f4f2599c4b0cbf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.554412 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4100abc9ff12471e9a4959b6b24f7e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.557724 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 57f522a825264ea7b215304dbb7aa2d5 (DEFAULT_TABLE table=default.Table_6a4dbd8e2b264bbca433946dfb1890b7 [id=9fec2fa31d474c5f8bc3b39041ed30bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.558493 1481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee3c521876324808857f7724028c895f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.558629 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c58e2439dc490c824b5f0023b5434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.558741 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 30bce965315d43948ecb8282c829c557 (DEFAULT_TABLE table=default.Table_98de64dffcd34afc91c487faa22cb705 [id=d9ae61ad6f8545d8a9945f08bdd731a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.559018 1625 tablet_service.cc:1505] Processing CreateTablet for tablet ccd46da86fe444baa675d3391cdd410c (DEFAULT_TABLE table=default.Table_a9c5f60f75d747b29b066b3e563b02a5 [id=3181d2379be74c45aa52ca84284b3475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.559448 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd46da86fe444baa675d3391cdd410c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.560779 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30bce965315d43948ecb8282c829c557. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.562654 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc7ccf1495924782b9b6ac1173bc73a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.565461 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 73278db2097d4001a538d62dacb97c9b (DEFAULT_TABLE table=default.Table_ba7eebe9102b4e27ad8227e35ed754ba [id=0568912ed6824b29bf68cd4f25ca2dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.565855 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73278db2097d4001a538d62dacb97c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.566387 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7be9290daabb404f955f05922d897fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.568840 1934 ts_manager.cc:194] Registered new tserver with Master: 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:02.570318 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_19fdf66f02124c62b4dad9bdeb7de7c4" } new_table_name: "default.Table_d47939bfce2049ccb93c29b51650629d" modify_external_catalogs: true
I20251024 19:48:02.561192 1616 tablet_service.cc:1505] Processing CreateTablet for tablet e38ef2ac19d14f9c91fa8abe86cc7987 (DEFAULT_TABLE table=default.Table_ea7e190995344cb8a1c35c95fc35b820 [id=50a4f64cc7504ea190832dd265c81caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.574899 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 607ae453221d422e9d07882830c02275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.575879 1351 tablet_service.cc:1505] Processing CreateTablet for tablet e38ef2ac19d14f9c91fa8abe86cc7987 (DEFAULT_TABLE table=default.Table_ea7e190995344cb8a1c35c95fc35b820 [id=50a4f64cc7504ea190832dd265c81caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.576192 1629 tablet_service.cc:1505] Processing CreateTablet for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 (DEFAULT_TABLE table=default.Table_7d5114465a524d37b479ed5a6cebb938 [id=fab966f6dc2c4b299c517ea170531ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.576315 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38ef2ac19d14f9c91fa8abe86cc7987. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.579926 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38ef2ac19d14f9c91fa8abe86cc7987. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.580353 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1cdd5ec32ca49c1bfafaf3a24050e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.581696 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 80c58e2439dc490c824b5f0023b5434b (DEFAULT_TABLE table=default.Table_2aa9405152c14d538c7164b50b336c7f [id=c00489e309ba4ada921f101a95bc8b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.582151 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c58e2439dc490c824b5f0023b5434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.582880 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet b79801594e95434cbb8fe7b75b14602e from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:02.575160 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bf08ccf7054fedb9efcbb00354c71a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.577651 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 769f576810ab49368eb0fc7837b7da58 (DEFAULT_TABLE table=default.Table_b905a1285caa407e9fb08e81639c9bc8 [id=1da47e6010f548d1b3d8c538150a8a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.588220 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 80c58e2439dc490c824b5f0023b5434b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.583297 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 58a8f4f90f794e83a158e0026e5cea7f (DEFAULT_TABLE table=default.Table_8255c10f050f4670a31e7c175a36cb62 [id=3fe81a9c4a134e908f4f2599c4b0cbf4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:02.589083 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc7ccf1495924782b9b6ac1173bc73a1 consensus state has no leader
I20251024 19:48:02.587267 1774 ts_tablet_manager.cc:1929] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251024 19:48:02.592247 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 57f522a825264ea7b215304dbb7aa2d5 (DEFAULT_TABLE table=default.Table_6a4dbd8e2b264bbca433946dfb1890b7 [id=9fec2fa31d474c5f8bc3b39041ed30bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.593816 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b79801594e95434cbb8fe7b75b14602e
I20251024 19:48:02.594547 1188 catalog_manager.cc:6719] ReplaceTablet: tablet b79801594e95434cbb8fe7b75b14602e deleted and replaced by tablet 3e6f8764e9ae472b9ac35a032182aed1
W20251024 19:48:02.595615 1305 leader_election.cc:343] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:02.597854 1345 tablet_service.cc:1505] Processing CreateTablet for tablet ccd46da86fe444baa675d3391cdd410c (DEFAULT_TABLE table=default.Table_a9c5f60f75d747b29b066b3e563b02a5 [id=3181d2379be74c45aa52ca84284b3475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.598516 1347 tablet_service.cc:1505] Processing CreateTablet for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 (DEFAULT_TABLE table=default.Table_7d5114465a524d37b479ed5a6cebb938 [id=fab966f6dc2c4b299c517ea170531ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.598912 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 769f576810ab49368eb0fc7837b7da58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.602730 1768 raft_consensus.cc:359] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.603034 1768 raft_consensus.cc:385] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:02.603648 1768 raft_consensus.cc:740] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:02.604473 1768 consensus_queue.cc:260] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.604888 2048 raft_consensus.cc:493] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.604919 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet b79801594e95434cbb8fe7b75b14602e with delete_type TABLET_DATA_DELETED (replaced by tablet 3e6f8764e9ae472b9ac35a032182aed1) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.605182 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58a8f4f90f794e83a158e0026e5cea7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.605085 2048 raft_consensus.cc:515] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.606061 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 73278db2097d4001a538d62dacb97c9b (DEFAULT_TABLE table=default.Table_ba7eebe9102b4e27ad8227e35ed754ba [id=0568912ed6824b29bf68cd4f25ca2dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.606565 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f522a825264ea7b215304dbb7aa2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.609476 2048 leader_election.cc:290] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:02.609994 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58a8f4f90f794e83a158e0026e5cea7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.613653 1774 log.cc:1199] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/f1ad65fd080e452e8094975d420336c7
I20251024 19:48:02.614887 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b9b1e0338fd14634a1e921735ae61289" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:02.615263 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73278db2097d4001a538d62dacb97c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.616209 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 30bce965315d43948ecb8282c829c557 (DEFAULT_TABLE table=default.Table_98de64dffcd34afc91c487faa22cb705 [id=d9ae61ad6f8545d8a9945f08bdd731a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.616714 1371 raft_consensus.cc:2468] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:02.617015 1774 ts_tablet_manager.cc:1950] T f1ad65fd080e452e8094975d420336c7 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:02.617869 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 220) took 2755 ms. Trace:
I20251024 19:48:02.618218 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_bbd727eca62c408ca76741ebea86eb3b" } new_table_name: "default.Table_b5a66c4e9cd8408782690b66cdb87473" modify_external_catalogs: true
I20251024 19:48:02.620468 1586 leader_election.cc:304] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:02.622613 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f522a825264ea7b215304dbb7aa2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.623546 2048 raft_consensus.cc:2804] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:02.623893 2048 raft_consensus.cc:493] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:02.624017 2048 raft_consensus.cc:3060] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.624266 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd46da86fe444baa675d3391cdd410c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.625886 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet b79801594e95434cbb8fe7b75b14602e with delete_type TABLET_DATA_DELETED (replaced by tablet 3e6f8764e9ae472b9ac35a032182aed1) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.627393 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1cdd5ec32ca49c1bfafaf3a24050e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.629326 1774 rpcz_store.cc:276] 1024 19:47:59.862099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.862303 (+ 204us) service_pool.cc:225] Handling call
1024 19:48:02.471416 (+2609113us) ts_tablet_manager.cc:1151] Deleting tablet f1ad65fd080e452e8094975d420336c7
1024 19:48:02.471424 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.498538 (+ 27114us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:02.574135 (+ 75597us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:02.617314 (+ 43179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2608861}
I20251024 19:48:02.630535 2048 raft_consensus.cc:515] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.631852 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet f1ad65fd080e452e8094975d420336c7 (table default.Table_a54ffafa18b848b79a290f58c9754bec [id=2fd5c541b06a46469e65ed0942d2c394]) successfully deleted
I20251024 19:48:02.632194 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 222) took 2760 ms. Trace:
I20251024 19:48:02.632292 1774 rpcz_store.cc:276] 1024 19:47:59.872024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.872090 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:02.631079 (+2758989us) ts_tablet_manager.cc:1151] Deleting tablet aebf76d4a66947ed9756643740418798
1024 19:48:02.631093 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.632130 (+ 1037us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2746249}
I20251024 19:48:02.632694 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 223) took 2755 ms. Trace:
W20251024 19:48:02.633216 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet aebf76d4a66947ed9756643740418798 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aebf76d4a66947ed9756643740418798 already in progress: creating tablet
I20251024 19:48:02.633991 2048 leader_election.cc:290] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:02.635985 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81a3de69a13c459b9b7c02761ab7261e consensus state has no leader
I20251024 19:48:02.637344 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30bce965315d43948ecb8282c829c557. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.637424 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b9b1e0338fd14634a1e921735ae61289" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:02.637681 1371 raft_consensus.cc:3060] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.632802 1774 rpcz_store.cc:276] 1024 19:47:59.877542 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.877613 (+ 71us) service_pool.cc:225] Handling call
1024 19:48:02.632540 (+2754927us) ts_tablet_manager.cc:1151] Deleting tablet 27b32126d04e480f8db7b2b20aba5873
1024 19:48:02.632550 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.632644 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2747581}
I20251024 19:48:02.646306 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8b8ee145388d4588b50306be38739650" } modify_external_catalogs: true
I20251024 19:48:02.646230 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_546f0aaae7694ef9807723a9ecc39602"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.646508 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 27b32126d04e480f8db7b2b20aba5873 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27b32126d04e480f8db7b2b20aba5873 already in progress: creating tablet
I20251024 19:48:02.646606 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8b8ee145388d4588b50306be38739650" } modify_external_catalogs: true
W20251024 19:48:02.646896 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_546f0aaae7694ef9807723a9ecc39602 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.647114 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ce25ea67811e4818be50de93930ab68e" } new_table_name: "default.Table_f203b363de4c47fca0a0599140d18279" modify_external_catalogs: true
I20251024 19:48:02.647359 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2919f656bb2c466ab7679bc4a465c7ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.647903 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2919f656bb2c466ab7679bc4a465c7ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.648399 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c4882268578941ccaf2c28be3074fdd2" } modify_external_catalogs: true
I20251024 19:48:02.648586 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c4882268578941ccaf2c28be3074fdd2" } modify_external_catalogs: true
I20251024 19:48:02.650838 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f4e511aa3c4b46479c523daa4149707f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.651330 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4e511aa3c4b46479c523daa4149707f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.652415 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3cd13b4df8f54c22aad4df0a6ccaef3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.652861 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cd13b4df8f54c22aad4df0a6ccaef3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.654472 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_964aae2f085749baaf4f7d0ee3ebb1f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.654887 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_964aae2f085749baaf4f7d0ee3ebb1f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.657470 1184 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 2346c03802a245a2af8c086b626d660d (table default.Table_03760ade20c7464f81dd428a3eb734fb [id=aadca5a1b6d4494aa37f8a8cc76ad12a]). Expected version 5 got 0
I20251024 19:48:02.673522 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4a69c0918d2e4eed9d91b1913ec2c21e
I20251024 19:48:02.674489 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2e7b3288182744969e217d4f174fe4ea
I20251024 19:48:02.678364 1346 tablet_service.cc:1505] Processing CreateTablet for tablet ab488e4971874a16b2e052c84c250777 (DEFAULT_TABLE table=default.Table_9ea4b3e0bf8d41ce844677ff6dc9ddfe [id=c69469ba6a9745f6a66092a678a5861d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.674584 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 224) took 2774 ms. Trace:
I20251024 19:48:02.682174 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 786fe2fd6fca407e83a76f74f96a8b16 (DEFAULT_TABLE table=default.Table_03318e0d18d34db48da05226fd40029a [id=f86c9d168b0e452899202f63a090f87a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:02.682608 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce0ef4443e574846a9f48fef3b893352 consensus state has no leader
I20251024 19:48:02.684083 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 292c31e7d4ce411bb336a1405ca43ba5 (DEFAULT_TABLE table=default.Table_13f216b61d134e4cba15849361a28394 [id=13f16de0ad1f43ac80639d3cf35fbb16]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.685148 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab488e4971874a16b2e052c84c250777. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.687773 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 786fe2fd6fca407e83a76f74f96a8b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.686929 1774 rpcz_store.cc:276] 1024 19:47:59.879403 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.879481 (+ 78us) service_pool.cc:225] Handling call
1024 19:48:02.645927 (+2766446us) ts_tablet_manager.cc:1151] Deleting tablet 50d919a0b3d74e9a97d136a84d50e24b
1024 19:48:02.645940 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.654029 (+ 8089us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2760795}
I20251024 19:48:02.689386 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292c31e7d4ce411bb336a1405ca43ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.687796 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 83b91a5d764449d5b82ef6d1ee0360f5 (DEFAULT_TABLE table=default.Table_325e96b0d32e4a3aab388a916c29ec26 [id=c6b5c56913684d08b89fde458e31b8cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.690418 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 780442b310e6409498e6bbe4256280a4 (DEFAULT_TABLE table=default.Table_7966ff9879d24254aed39562b2f127d7 [id=45b38d5e9fd6472fa318b41863080708]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.693171 1339 tablet_service.cc:1505] Processing CreateTablet for tablet c1fc0f99ca5c42bfb9663e21a7b761c9 (DEFAULT_TABLE table=default.Table_8631b5d780614ecdb746a74a4d97b8b5 [id=a32bd550682746f0a2861ff8be404992]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.693881 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_be61dd09c4c94f568d723c4bace3a038" } modify_external_catalogs: true
I20251024 19:48:02.694087 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_be61dd09c4c94f568d723c4bace3a038" } modify_external_catalogs: true
I20251024 19:48:02.695839 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a1d09325af844d71b290d95b8a4aaf12" } modify_external_catalogs: true
I20251024 19:48:02.696120 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet d76410afd7ba4c73836f4791ef739056 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:02.696096 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a1d09325af844d71b290d95b8a4aaf12" } modify_external_catalogs: true
I20251024 19:48:02.698457 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 2b2992a738b440d798f5e61cc45ec088 (DEFAULT_TABLE table=default.Table_405cbdb6a024494a81e147548e24d436 [id=40394401a8a84431bfb5e2b1db774832]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.698662 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83b91a5d764449d5b82ef6d1ee0360f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.711124 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a69c0918d2e4eed9d91b1913ec2c21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.713976 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_50d055bf7fc34e8fa68f1a188aea6218"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.714859 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50d055bf7fc34e8fa68f1a188aea6218 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.715945 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2952ae2bfd4d4924afd95ddf2e8404bd" } new_table_name: "default.Table_9e14814a2db64e0b9602e41f926f3b62" modify_external_catalogs: true
I20251024 19:48:02.718112 2158 raft_consensus.cc:2804] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:02.720391 1337 tablet_service.cc:1505] Processing CreateTablet for tablet e0d5d210d8d8499cbfdbdaa29e5bb40b (DEFAULT_TABLE table=default.Table_8b66ef90eef047bfbf665db21ee48274 [id=a2adb68f20424f9a8b8d8785496b2f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.725204 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 225) took 2812 ms. Trace:
I20251024 19:48:02.725395 1774 rpcz_store.cc:276] 1024 19:47:59.880203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.880253 (+ 50us) service_pool.cc:225] Handling call
1024 19:48:02.689490 (+2809237us) ts_tablet_manager.cc:1151] Deleting tablet b88a6a5237d146f5b6f082ca3af25b2b
1024 19:48:02.689501 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.692607 (+ 3106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2804144}
I20251024 19:48:02.727917 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1fc0f99ca5c42bfb9663e21a7b761c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.727950 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2919f656bb2c466ab7679bc4a465c7ea" } modify_external_catalogs: true
I20251024 19:48:02.727946 1620 tablet_service.cc:1505] Processing CreateTablet for tablet ab488e4971874a16b2e052c84c250777 (DEFAULT_TABLE table=default.Table_9ea4b3e0bf8d41ce844677ff6dc9ddfe [id=c69469ba6a9745f6a66092a678a5861d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.728128 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2919f656bb2c466ab7679bc4a465c7ea" } modify_external_catalogs: true
I20251024 19:48:02.728188 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0d5d210d8d8499cbfdbdaa29e5bb40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.728838 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab488e4971874a16b2e052c84c250777. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.729354 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 780442b310e6409498e6bbe4256280a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.730324 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b2992a738b440d798f5e61cc45ec088. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.730768 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 83b91a5d764449d5b82ef6d1ee0360f5 (DEFAULT_TABLE table=default.Table_325e96b0d32e4a3aab388a916c29ec26 [id=c6b5c56913684d08b89fde458e31b8cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.731627 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83b91a5d764449d5b82ef6d1ee0360f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.733127 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e7b3288182744969e217d4f174fe4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.733289 1371 raft_consensus.cc:2468] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:02.735267 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ba89fbfa74a0425c9db8f48b5f94eb83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:02.735740 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d47939bfce2049ccb93c29b51650629d" } new_table_name: "default.Table_6a5150d7236d4d558c9f9fb4e70a78dd" modify_external_catalogs: true
I20251024 19:48:02.735882 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d5aeb6bab67b4c1ea34aa869731de12a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:02.736181 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e25a14d3a9674ad38cce670621e477df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:02.736393 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5aeb6bab67b4c1ea34aa869731de12a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:02.736671 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e25a14d3a9674ad38cce670621e477df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:02.735882 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba89fbfa74a0425c9db8f48b5f94eb83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:02.733588 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 226) took 2846 ms. Trace:
I20251024 19:48:02.738020 1774 rpcz_store.cc:276] 1024 19:47:59.881005 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.881070 (+ 65us) service_pool.cc:225] Handling call
1024 19:48:02.725889 (+2844819us) ts_tablet_manager.cc:1151] Deleting tablet fca73ab61f0c47df9186459caf2b9a32
1024 19:48:02.725901 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.727263 (+ 1362us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2840502}
I20251024 19:48:02.738495 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 227) took 2853 ms. Trace:
I20251024 19:48:02.738598 1774 rpcz_store.cc:276] 1024 19:47:59.884571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.884629 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:02.738321 (+2853692us) ts_tablet_manager.cc:1151] Deleting tablet 44ee03ede02c4bdab706fde910978332
1024 19:48:02.738330 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.738428 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2852762}
I20251024 19:48:02.739146 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 221) took 2869 ms. Trace:
I20251024 19:48:02.739364 1774 rpcz_store.cc:276] 1024 19:47:59.869666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:47:59.869762 (+ 96us) service_pool.cc:225] Handling call
1024 19:48:02.738833 (+2869071us) ts_tablet_manager.cc:1151] Deleting tablet 2037fcca87b94b5fb31639b4d07307b0
1024 19:48:02.738842 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.738937 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2853043}
I20251024 19:48:02.739500 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 292c31e7d4ce411bb336a1405ca43ba5 (DEFAULT_TABLE table=default.Table_13f216b61d134e4cba15849361a28394 [id=13f16de0ad1f43ac80639d3cf35fbb16]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.739914 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 230) took 2728 ms. Trace:
I20251024 19:48:02.740080 1774 rpcz_store.cc:276] 1024 19:48:00.011798 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.011857 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:02.739602 (+2727745us) ts_tablet_manager.cc:1151] Deleting tablet 0a12e415590a4c71aab4ad7d30412c83
1024 19:48:02.739610 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.739756 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2726258}
W20251024 19:48:02.742372 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 50d919a0b3d74e9a97d136a84d50e24b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50d919a0b3d74e9a97d136a84d50e24b already in progress: creating tablet
W20251024 19:48:02.742954 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet fca73ab61f0c47df9186459caf2b9a32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fca73ab61f0c47df9186459caf2b9a32 already in progress: creating tablet
W20251024 19:48:02.743191 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 44ee03ede02c4bdab706fde910978332 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ee03ede02c4bdab706fde910978332 already in progress: creating tablet
I20251024 19:48:02.743459 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a69c0918d2e4eed9d91b1913ec2c21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:02.743927 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 2b2992a738b440d798f5e61cc45ec088 (DEFAULT_TABLE table=default.Table_405cbdb6a024494a81e147548e24d436 [id=40394401a8a84431bfb5e2b1db774832]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.743753 1617 tablet_service.cc:1505] Processing CreateTablet for tablet c1fc0f99ca5c42bfb9663e21a7b761c9 (DEFAULT_TABLE table=default.Table_8631b5d780614ecdb746a74a4d97b8b5 [id=a32bd550682746f0a2861ff8be404992]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.744095 2158 raft_consensus.cc:697] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:02.747941 1768 ts_tablet_manager.cc:1434] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.146s user 0.004s sys 0.004s
I20251024 19:48:02.748414 1586 leader_election.cc:304] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:02.748791 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e7b3288182744969e217d4f174fe4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:02.748875 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 0a12e415590a4c71aab4ad7d30412c83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a12e415590a4c71aab4ad7d30412c83 already in progress: creating tablet
I20251024 19:48:02.749890 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 780442b310e6409498e6bbe4256280a4 (DEFAULT_TABLE table=default.Table_7966ff9879d24254aed39562b2f127d7 [id=45b38d5e9fd6472fa318b41863080708]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.751336 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 4a69c0918d2e4eed9d91b1913ec2c21e (DEFAULT_TABLE table=default.Table_8b8ee145388d4588b50306be38739650 [id=8f925cc7435543aa8aacba668bd66b52]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.752494 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 4a69c0918d2e4eed9d91b1913ec2c21e (DEFAULT_TABLE table=default.Table_8b8ee145388d4588b50306be38739650 [id=8f925cc7435543aa8aacba668bd66b52]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.753229 1628 tablet_service.cc:1505] Processing CreateTablet for tablet e0d5d210d8d8499cbfdbdaa29e5bb40b (DEFAULT_TABLE table=default.Table_8b66ef90eef047bfbf665db21ee48274 [id=a2adb68f20424f9a8b8d8785496b2f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.757520 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 64499c8734d34c66b0b52a2cd3ef0673 (DEFAULT_TABLE table=default.Table_511ba1e13b8f43e299835a4c30063587 [id=b76ea2d1917f4b3cb23721c0df6d641f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.760087 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b2992a738b440d798f5e61cc45ec088. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.763953 2020 raft_consensus.cc:493] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.764223 2020 raft_consensus.cc:515] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.765081 2020 leader_election.cc:290] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:02.766096 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c58e2439dc490c824b5f0023b5434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.770123 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b9b1e0338fd14634a1e921735ae61289" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:02.772940 1511 raft_consensus.cc:2468] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:02.770426 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b9b1e0338fd14634a1e921735ae61289" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:02.773753 1510 raft_consensus.cc:3060] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.774343 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 769f576810ab49368eb0fc7837b7da58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.774408 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 231) took 2737 ms. Trace:
I20251024 19:48:02.774542 1774 rpcz_store.cc:276] 1024 19:48:00.013520 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.013590 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:02.749330 (+2735740us) ts_tablet_manager.cc:1151] Deleting tablet a76d61c3464b41519d7657796bd853ff
1024 19:48:02.749341 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.751010 (+ 1669us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2735542}
W20251024 19:48:02.775054 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet a76d61c3464b41519d7657796bd853ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a76d61c3464b41519d7657796bd853ff already in progress: creating tablet
I20251024 19:48:02.775084 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 30bce965315d43948ecb8282c829c557 (DEFAULT_TABLE table=default.Table_98de64dffcd34afc91c487faa22cb705 [id=d9ae61ad6f8545d8a9945f08bdd731a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.775539 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f522a825264ea7b215304dbb7aa2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.775451 2158 consensus_queue.cc:237] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:02.776904 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58a8f4f90f794e83a158e0026e5cea7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.777428 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38ef2ac19d14f9c91fa8abe86cc7987. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.777539 1478 tablet_service.cc:1505] Processing CreateTablet for tablet ccd46da86fe444baa675d3391cdd410c (DEFAULT_TABLE table=default.Table_a9c5f60f75d747b29b066b3e563b02a5 [id=3181d2379be74c45aa52ca84284b3475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.777709 2146 raft_consensus.cc:2804] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:02.778230 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 73278db2097d4001a538d62dacb97c9b (DEFAULT_TABLE table=default.Table_ba7eebe9102b4e27ad8227e35ed754ba [id=0568912ed6824b29bf68cd4f25ca2dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.778527 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b9b1e0338fd14634a1e921735ae61289" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:02.779340 1477 tablet_service.cc:1505] Processing CreateTablet for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 (DEFAULT_TABLE table=default.Table_7d5114465a524d37b479ed5a6cebb938 [id=fab966f6dc2c4b299c517ea170531ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.780555 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet b79801594e95434cbb8fe7b75b14602e with delete_type TABLET_DATA_DELETED (replaced by tablet 3e6f8764e9ae472b9ac35a032182aed1) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.781121 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 9623d7baf0174a019e2353a5effef081 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.781648 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a69c0918d2e4eed9d91b1913ec2c21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.782083 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 786fe2fd6fca407e83a76f74f96a8b16 (DEFAULT_TABLE table=default.Table_03318e0d18d34db48da05226fd40029a [id=f86c9d168b0e452899202f63a090f87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.782828 1475 tablet_service.cc:1505] Processing CreateTablet for tablet ab488e4971874a16b2e052c84c250777 (DEFAULT_TABLE table=default.Table_9ea4b3e0bf8d41ce844677ff6dc9ddfe [id=c69469ba6a9745f6a66092a678a5861d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.783282 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 64499c8734d34c66b0b52a2cd3ef0673 (DEFAULT_TABLE table=default.Table_511ba1e13b8f43e299835a4c30063587 [id=b76ea2d1917f4b3cb23721c0df6d641f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.783738 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 292c31e7d4ce411bb336a1405ca43ba5 (DEFAULT_TABLE table=default.Table_13f216b61d134e4cba15849361a28394 [id=13f16de0ad1f43ac80639d3cf35fbb16]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.784637 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 83b91a5d764449d5b82ef6d1ee0360f5 (DEFAULT_TABLE table=default.Table_325e96b0d32e4a3aab388a916c29ec26 [id=c6b5c56913684d08b89fde458e31b8cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.785475 1472 tablet_service.cc:1505] Processing CreateTablet for tablet c1fc0f99ca5c42bfb9663e21a7b761c9 (DEFAULT_TABLE table=default.Table_8631b5d780614ecdb746a74a4d97b8b5 [id=a32bd550682746f0a2861ff8be404992]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.789000 1768 tablet_bootstrap.cc:492] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:02.790051 2146 raft_consensus.cc:697] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:02.790606 2146 consensus_queue.cc:237] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.797793 1490 tablet_service.cc:1505] Processing CreateTablet for tablet e0d5d210d8d8499cbfdbdaa29e5bb40b (DEFAULT_TABLE table=default.Table_8b66ef90eef047bfbf665db21ee48274 [id=a2adb68f20424f9a8b8d8785496b2f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.798200 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0d5d210d8d8499cbfdbdaa29e5bb40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.798977 1480 tablet_service.cc:1505] Processing CreateTablet for tablet 780442b310e6409498e6bbe4256280a4 (DEFAULT_TABLE table=default.Table_7966ff9879d24254aed39562b2f127d7 [id=45b38d5e9fd6472fa318b41863080708]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.799315 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 780442b310e6409498e6bbe4256280a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.800305 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30bce965315d43948ecb8282c829c557. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.800832 1478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd46da86fe444baa675d3391cdd410c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.801379 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73278db2097d4001a538d62dacb97c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.801935 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1cdd5ec32ca49c1bfafaf3a24050e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.802549 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 786fe2fd6fca407e83a76f74f96a8b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.803174 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab488e4971874a16b2e052c84c250777. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.803794 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292c31e7d4ce411bb336a1405ca43ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.804754 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83b91a5d764449d5b82ef6d1ee0360f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.805401 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1fc0f99ca5c42bfb9663e21a7b761c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.822266 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292c31e7d4ce411bb336a1405ca43ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.826148 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b9b1e0338fd14634a1e921735ae61289" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:02.828819 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0d5d210d8d8499cbfdbdaa29e5bb40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.829932 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64499c8734d34c66b0b52a2cd3ef0673. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.831530 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1fc0f99ca5c42bfb9663e21a7b761c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.835475 1371 raft_consensus.cc:2393] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10a9536207e343f188e0ef709081d04e in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:02.836721 1510 raft_consensus.cc:2468] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:02.843261 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 243) took 2583 ms. Trace:
I20251024 19:48:02.844440 1472 tablet_service.cc:1505] Processing CreateTablet for tablet 2b2992a738b440d798f5e61cc45ec088 (DEFAULT_TABLE table=default.Table_405cbdb6a024494a81e147548e24d436 [id=40394401a8a84431bfb5e2b1db774832]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.847313 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 780442b310e6409498e6bbe4256280a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.854025 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 786fe2fd6fca407e83a76f74f96a8b16 (DEFAULT_TABLE table=default.Table_03318e0d18d34db48da05226fd40029a [id=f86c9d168b0e452899202f63a090f87a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.860392 2154 raft_consensus.cc:493] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.861466 1774 rpcz_store.cc:276] 1024 19:48:00.217331 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.217377 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:02.775382 (+2558005us) ts_tablet_manager.cc:1151] Deleting tablet b1fa439afe5a48a79557019687319af6
1024 19:48:02.797357 (+ 21975us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.800412 (+ 3055us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2557716}
W20251024 19:48:02.864214 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b1fa439afe5a48a79557019687319af6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1fa439afe5a48a79557019687319af6 already in progress: creating tablet
I20251024 19:48:02.864850 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64499c8734d34c66b0b52a2cd3ef0673. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.872649 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a69c0918d2e4eed9d91b1913ec2c21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.860584 2154 raft_consensus.cc:515] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.878783 2020 raft_consensus.cc:493] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.879069 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b2992a738b440d798f5e61cc45ec088. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.880136 2020 raft_consensus.cc:515] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.880520 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 242) took 2656 ms. Trace:
I20251024 19:48:02.880690 1774 rpcz_store.cc:276] 1024 19:48:00.217213 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.218038 (+ 825us) service_pool.cc:225] Handling call
1024 19:48:02.864672 (+2646634us) ts_tablet_manager.cc:1151] Deleting tablet c82f60819e7a442bbc0cc04659e14e87
1024 19:48:02.864683 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.873433 (+ 8750us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2646416}
I20251024 19:48:02.880893 2020 leader_election.cc:290] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:02.881521 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet c82f60819e7a442bbc0cc04659e14e87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c82f60819e7a442bbc0cc04659e14e87 already in progress: creating tablet
I20251024 19:48:02.884249 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9623d7baf0174a019e2353a5effef081" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:02.887244 1444 leader_election.cc:343] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:02.887856 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 253) took 2537 ms. Trace:
I20251024 19:48:02.887990 1774 rpcz_store.cc:276] 1024 19:48:00.346009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.346462 (+ 453us) service_pool.cc:225] Handling call
1024 19:48:02.881153 (+2534691us) ts_tablet_manager.cc:1151] Deleting tablet 3887fc21726846279fecb02672b9bc07
1024 19:48:02.881165 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.883081 (+ 1916us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2526697}
I20251024 19:48:02.888470 1481 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a69c0918d2e4eed9d91b1913ec2c21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:02.888741 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3887fc21726846279fecb02672b9bc07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3887fc21726846279fecb02672b9bc07 already in progress: creating tablet
I20251024 19:48:02.889077 1768 tablet_bootstrap.cc:654] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:02.890802 1774 tablet_replica.cc:333] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:02.891036 1774 raft_consensus.cc:2243] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:02.906968 2154 leader_election.cc:290] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:02.910998 1774 raft_consensus.cc:2272] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:02.913371 1478 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e7b3288182744969e217d4f174fe4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:02.915704 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f05ad092936b4bafa3ae795904c2df78" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:02.916107 1650 raft_consensus.cc:2468] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:02.916828 1305 leader_election.cc:304] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:02.917009 1767 tablet_bootstrap.cc:654] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:02.917129 2154 raft_consensus.cc:2804] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:02.917320 2154 raft_consensus.cc:493] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:02.917425 2154 raft_consensus.cc:3060] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:02.917604 1447 leader_election.cc:343] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 9623d7baf0174a019e2353a5effef081
I20251024 19:48:02.916721 1774 ts_tablet_manager.cc:1916] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:02.918079 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 64499c8734d34c66b0b52a2cd3ef0673 (DEFAULT_TABLE table=default.Table_511ba1e13b8f43e299835a4c30063587 [id=b76ea2d1917f4b3cb23721c0df6d641f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.918169 1776 ts_tablet_manager.cc:1929] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:02.918524 2020 consensus_queue.cc:1048] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20251024 19:48:02.919202 1766 tablet_bootstrap.cc:492] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:02.920909 2154 raft_consensus.cc:515] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.929690 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 786fe2fd6fca407e83a76f74f96a8b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.930868 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f05ad092936b4bafa3ae795904c2df78" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:02.931087 1650 raft_consensus.cc:3060] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.934844 1776 log.cc:1199] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/90405a0c13bd4c229b011c90232f751d
I20251024 19:48:02.935626 1776 ts_tablet_manager.cc:1950] T 90405a0c13bd4c229b011c90232f751d P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:02.922587 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 4a69c0918d2e4eed9d91b1913ec2c21e (DEFAULT_TABLE table=default.Table_8b8ee145388d4588b50306be38739650 [id=8f925cc7435543aa8aacba668bd66b52]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:02.923532 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f05ad092936b4bafa3ae795904c2df78" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:02.923619 1444 leader_election.cc:304] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:02.923712 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64499c8734d34c66b0b52a2cd3ef0673. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.927028 1447 leader_election.cc:304] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:02.935699 2154 leader_election.cc:290] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:02.941139 1510 raft_consensus.cc:2468] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:02.941143 2141 raft_consensus.cc:2749] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:02.941749 1766 ts_tablet_manager.cc:1403] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.507s user 0.013s sys 0.018s
I20251024 19:48:02.944722 2020 raft_consensus.cc:2749] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:02.947904 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 328) took 1309 ms. Trace:
I20251024 19:48:02.948515 1776 rpcz_store.cc:276] 1024 19:48:01.638381 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.670878 (+ 32497us) service_pool.cc:225] Handling call
1024 19:48:02.308552 (+637674us) ts_tablet_manager.cc:1151] Deleting tablet 90405a0c13bd4c229b011c90232f751d
1024 19:48:02.308562 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.705437 (+396875us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:02.879056 (+173619us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:02.947793 (+ 68737us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":599985}
I20251024 19:48:02.949719 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 90405a0c13bd4c229b011c90232f751d (table default.Table_eb9c8019451d46fba31a87a0b6ac6ad8 [id=e5b1dd31c2114e8a95ba33981029255c]) successfully deleted
I20251024 19:48:02.950606 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 330) took 1243 ms. Trace:
I20251024 19:48:02.950942 1776 rpcz_store.cc:276] 1024 19:48:01.706765 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.707815 (+ 1050us) service_pool.cc:225] Handling call
1024 19:48:02.950207 (+1242392us) ts_tablet_manager.cc:1151] Deleting tablet dc1b6f0a377945089c50012f74d38564
1024 19:48:02.950222 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.950490 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164119}
I20251024 19:48:02.958334 1768 tablet_bootstrap.cc:492] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:02.958513 1768 ts_tablet_manager.cc:1403] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.210s user 0.003s sys 0.007s
I20251024 19:48:02.959285 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a69c0918d2e4eed9d91b1913ec2c21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:02.959343 1768 raft_consensus.cc:359] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:02.959532 1768 raft_consensus.cc:385] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:02.959661 1768 raft_consensus.cc:740] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:02.960191 1768 consensus_queue.cc:260] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:02.960532 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f05ad092936b4bafa3ae795904c2df78" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:02.960966 1768 ts_tablet_manager.cc:1434] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:02.961761 1511 raft_consensus.cc:3060] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:02.962213 1650 raft_consensus.cc:2468] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:02.962766 1305 leader_election.cc:304] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:02.963454 2154 raft_consensus.cc:2804] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:02.963649 2154 raft_consensus.cc:697] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:02.963979 2154 consensus_queue.cc:237] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:02.987851 1333 tablet_service.cc:1552] Processing DeleteTablet for tablet 897db285f5c84930a5b438be1cb4eb96 with delete_type TABLET_DATA_DELETED (replaced by tablet 758df87acbdb4adaa83e1ecbc44e7cf1) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:02.989081 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c8ab8e29b5a94252a5395f467f5e130f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:02.990211 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8255c10f050f4670a31e7c175a36cb62" } new_table_name: "default.Table_c7b7dc69f2e640f6bb21208e8d8f32b8" modify_external_catalogs: true
I20251024 19:48:02.993875 1768 tablet_bootstrap.cc:492] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:02.994691 2158 raft_consensus.cc:493] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:02.994849 2158 raft_consensus.cc:515] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:02.995721 2158 leader_election.cc:290] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:02.996421 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31647a48fb834c80b6920c756a94250d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:02.996747 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8ab8e29b5a94252a5395f467f5e130f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:02.996927 1305 leader_election.cc:343] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:03.000087 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9026d9e801294dd2bfdb9c90e45a0990 consensus state has no leader
I20251024 19:48:03.000749 1511 raft_consensus.cc:2468] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:03.001925 1186 catalog_manager.cc:5649] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43bec2f03fc849099e254f6335e9d532 (127.0.252.193). New cstate: current_term: 1 leader_uuid: "43bec2f03fc849099e254f6335e9d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:03.002085 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31647a48fb834c80b6920c756a94250d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:03.002818 1308 leader_election.cc:343] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.002992 1308 leader_election.cc:304] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:03.003074 1186 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 2346c03802a245a2af8c086b626d660d (table default.Table_03760ade20c7464f81dd428a3eb734fb [id=aadca5a1b6d4494aa37f8a8cc76ad12a]). Expected version 5 got 0
I20251024 19:48:03.003543 2158 raft_consensus.cc:2749] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:03.004290 1768 tablet_bootstrap.cc:654] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.004891 1766 raft_consensus.cc:359] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:03.005852 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc7ccf1495924782b9b6ac1173bc73a1 consensus state has no leader
I20251024 19:48:03.006162 1766 raft_consensus.cc:385] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.006247 1766 raft_consensus.cc:740] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.006242 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_873d6a3a39344554b0ca73c575129c36" } new_table_name: "default.Table_5d986b803bac48beb19d3de7faf77cdf" modify_external_catalogs: true
I20251024 19:48:03.010040 1774 ts_tablet_manager.cc:1929] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:03.010177 1774 log.cc:1199] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/369cebfda6ee43638b79f8652fa55855
I20251024 19:48:03.010176 1767 tablet_bootstrap.cc:492] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:03.010349 1767 ts_tablet_manager.cc:1403] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.497s user 0.002s sys 0.004s
I20251024 19:48:03.010697 1774 ts_tablet_manager.cc:1950] T 369cebfda6ee43638b79f8652fa55855 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:03.011126 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 251) took 2673 ms. Trace:
I20251024 19:48:03.011353 1767 raft_consensus.cc:359] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:03.011696 1767 raft_consensus.cc:385] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.011850 1767 raft_consensus.cc:740] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.012244 1767 consensus_queue.cc:260] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:03.012923 1767 ts_tablet_manager.cc:1434] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:03.013952 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 329) took 1364 ms. Trace:
I20251024 19:48:03.011233 1774 rpcz_store.cc:276] 1024 19:48:00.337383 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.337505 (+ 122us) service_pool.cc:225] Handling call
1024 19:48:02.888430 (+2550925us) ts_tablet_manager.cc:1151] Deleting tablet 369cebfda6ee43638b79f8652fa55855
1024 19:48:02.888445 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:02.961831 (+ 73386us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.009892 (+ 48061us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.011085 (+ 1193us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2532622}
I20251024 19:48:03.014972 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet bc31e6c25ad14947af20abd766f12e78
I20251024 19:48:03.015799 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_aa4ebf0155944ee2bfba2c23eacb5655"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.016448 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa4ebf0155944ee2bfba2c23eacb5655 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.016848 1774 tablet_replica.cc:333] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:03.017102 1774 raft_consensus.cc:2243] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:03.017961 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet dc14b0cdb7f342ac956a7857ab3f35d7
I20251024 19:48:03.019476 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ecf8ed863a5b41fb888927d73b57c8a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.020366 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecf8ed863a5b41fb888927d73b57c8a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.021739 1774 raft_consensus.cc:2272] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:03.022168 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet d76410afd7ba4c73836f4791ef739056
I20251024 19:48:03.022697 1183 catalog_manager.cc:6719] ReplaceTablet: tablet d76410afd7ba4c73836f4791ef739056 deleted and replaced by tablet b153f75caebb4d869e77529c9c9bb50d
I20251024 19:48:03.023063 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cb99a44a90e849be948f4a481c07a91d" } modify_external_catalogs: true
I20251024 19:48:03.023263 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cb99a44a90e849be948f4a481c07a91d" } modify_external_catalogs: true
I20251024 19:48:03.024006 2020 consensus_queue.cc:1048] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20251024 19:48:03.024106 1774 ts_tablet_manager.cc:1916] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:03.026360 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 58a8f4f90f794e83a158e0026e5cea7f consensus state has no leader
I20251024 19:48:03.026796 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c564241bf74c49deaa26052b5c1d5afe" } modify_external_catalogs: true
I20251024 19:48:03.026968 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c564241bf74c49deaa26052b5c1d5afe" } modify_external_catalogs: true
I20251024 19:48:03.028362 1766 consensus_queue.cc:260] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.031502 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_61a8b015b11f4effa0245b9737c3c2ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.032115 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61a8b015b11f4effa0245b9737c3c2ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.034082 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a1c0c56a2bd5452293c935a6aecb39e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:03.034371 1617 tablet_service.cc:1505] Processing CreateTablet for tablet b22b58404b8b441ba97ef536885a66e0 (DEFAULT_TABLE table=default.Table_f4e511aa3c4b46479c523daa4149707f [id=bdfe9c0f34ac41eebb843bd17fa9635c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:03.034614 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1c0c56a2bd5452293c935a6aecb39e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.034833 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b22b58404b8b441ba97ef536885a66e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.036923 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet d76410afd7ba4c73836f4791ef739056 with delete_type TABLET_DATA_DELETED (replaced by tablet b153f75caebb4d869e77529c9c9bb50d) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.037250 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet dc14b0cdb7f342ac956a7857ab3f35d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.037606 1766 ts_tablet_manager.cc:1434] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.087s user 0.002s sys 0.000s
I20251024 19:48:03.037962 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet bc31e6c25ad14947af20abd766f12e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.038168 1628 tablet_service.cc:1505] Processing CreateTablet for tablet ebcc09d8e029451d8fb6b6601ad16087 (DEFAULT_TABLE table=default.Table_546f0aaae7694ef9807723a9ecc39602 [id=3e8b487a98c64acd8c6c6dbb35197598]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.038561 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebcc09d8e029451d8fb6b6601ad16087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.040715 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 92d78642772d44179e67d75c51b9da4e (DEFAULT_TABLE table=default.Table_3cd13b4df8f54c22aad4df0a6ccaef3e [id=37332f920f4d4827b16f1c1c7bffda6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.041160 2174 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc8f71a5b70749a4a3e3a599bcb3d22c)
I20251024 19:48:03.041291 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92d78642772d44179e67d75c51b9da4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.042321 1333 tablet_service.cc:1505] Processing CreateTablet for tablet b22b58404b8b441ba97ef536885a66e0 (DEFAULT_TABLE table=default.Table_f4e511aa3c4b46479c523daa4149707f [id=bdfe9c0f34ac41eebb843bd17fa9635c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.042753 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b22b58404b8b441ba97ef536885a66e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.014160 1776 rpcz_store.cc:276] 1024 19:48:01.649313 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.688930 (+ 39617us) service_pool.cc:225] Handling call
1024 19:48:03.013680 (+1324750us) ts_tablet_manager.cc:1151] Deleting tablet 758df87acbdb4adaa83e1ecbc44e7cf1
1024 19:48:03.013694 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.013850 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1165386}
I20251024 19:48:03.046881 1346 tablet_service.cc:1505] Processing CreateTablet for tablet ebcc09d8e029451d8fb6b6601ad16087 (DEFAULT_TABLE table=default.Table_546f0aaae7694ef9807723a9ecc39602 [id=3e8b487a98c64acd8c6c6dbb35197598]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.046890 2174 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:48:03.047400 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebcc09d8e029451d8fb6b6601ad16087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.048307 2174 leader_election.cc:290] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82e82847f8904b59b03f446b7fce6233 (127.0.252.253:41375), dc8f71a5b70749a4a3e3a599bcb3d22c (127.0.252.252:36627)
I20251024 19:48:03.050674 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 3e6f8764e9ae472b9ac35a032182aed1 (DEFAULT_TABLE table=default.Table_aa7c283553a841e9ab2fa21fb6485a11 [id=7ec9391c52b1424eb52850f2c6ba4779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.051295 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet bc31e6c25ad14947af20abd766f12e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.051376 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet dc14b0cdb7f342ac956a7857ab3f35d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.051488 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e6f8764e9ae472b9ac35a032182aed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.052191 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet d76410afd7ba4c73836f4791ef739056 with delete_type TABLET_DATA_DELETED (replaced by tablet b153f75caebb4d869e77529c9c9bb50d) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:03.053898 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 758df87acbdb4adaa83e1ecbc44e7cf1 already in progress: creating tablet
I20251024 19:48:03.054514 1129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef8906492b1a4368936fac95c11e1f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 335 } } ignore_live_leader: false dest_uuid: "82e82847f8904b59b03f446b7fce6233" is_pre_election: true
I20251024 19:48:03.060173 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 92d78642772d44179e67d75c51b9da4e (DEFAULT_TABLE table=default.Table_3cd13b4df8f54c22aad4df0a6ccaef3e [id=37332f920f4d4827b16f1c1c7bffda6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.060735 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92d78642772d44179e67d75c51b9da4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.062403 1768 tablet_bootstrap.cc:492] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:03.062572 1768 ts_tablet_manager.cc:1403] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.002s
I20251024 19:48:03.063866 1768 raft_consensus.cc:359] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.064106 1768 raft_consensus.cc:385] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.064255 1768 raft_consensus.cc:740] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.064493 1768 consensus_queue.cc:260] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.065025 1768 ts_tablet_manager.cc:1434] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:03.065407 1768 tablet_bootstrap.cc:492] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:03.065384 1766 tablet_bootstrap.cc:492] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:03.074767 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 3e6f8764e9ae472b9ac35a032182aed1 (DEFAULT_TABLE table=default.Table_aa7c283553a841e9ab2fa21fb6485a11 [id=7ec9391c52b1424eb52850f2c6ba4779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.076009 1768 tablet_bootstrap.cc:654] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.077824 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 332) took 1289 ms. Trace:
I20251024 19:48:03.081115 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 0797d14e9fd846788164747523df0e2c (DEFAULT_TABLE table=default.Table_964aae2f085749baaf4f7d0ee3ebb1f4 [id=a07755e138c84fd18c0fe72e81ab758b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.081534 1629 tablet_service.cc:1505] Processing CreateTablet for tablet bc31e6c25ad14947af20abd766f12e78 (DEFAULT_TABLE table=default.Table_2919f656bb2c466ab7679bc4a465c7ea [id=e0487ce7599645a2914ca3c8c5238b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.083956 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e6f8764e9ae472b9ac35a032182aed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.085881 1767 tablet_bootstrap.cc:492] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:03.086130 1776 rpcz_store.cc:276] 1024 19:48:01.786837 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.786983 (+ 146us) service_pool.cc:225] Handling call
1024 19:48:03.046162 (+1259179us) ts_tablet_manager.cc:1151] Deleting tablet 6f630fd412ca4c4f8570359c1678e6af
1024 19:48:03.046175 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.075785 (+ 29610us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1256210}
I20251024 19:48:03.086658 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 333) took 1299 ms. Trace:
W20251024 19:48:03.087154 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 6f630fd412ca4c4f8570359c1678e6af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f630fd412ca4c4f8570359c1678e6af already in progress: creating tablet
I20251024 19:48:03.088536 1776 rpcz_store.cc:276] 1024 19:48:01.787518 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.792979 (+ 5461us) service_pool.cc:225] Handling call
1024 19:48:03.086470 (+1293491us) ts_tablet_manager.cc:1151] Deleting tablet d2a68bb1dc6c483388784b9bb19ce065
1024 19:48:03.086482 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.086609 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290333}
I20251024 19:48:03.088992 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 334) took 1293 ms. Trace:
I20251024 19:48:03.089402 1776 rpcz_store.cc:276] 1024 19:48:01.795757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.795806 (+ 49us) service_pool.cc:225] Handling call
1024 19:48:03.088835 (+1293029us) ts_tablet_manager.cc:1151] Deleting tablet fdfa5eea3b674a3389a156d1c72204df
1024 19:48:03.088845 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.088957 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1292146}
I20251024 19:48:03.089736 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc31e6c25ad14947af20abd766f12e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.089980 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0797d14e9fd846788164747523df0e2c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:03.094003 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d2a68bb1dc6c483388784b9bb19ce065 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2a68bb1dc6c483388784b9bb19ce065 already in progress: creating tablet
W20251024 19:48:03.094368 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet fdfa5eea3b674a3389a156d1c72204df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdfa5eea3b674a3389a156d1c72204df already in progress: creating tablet
I20251024 19:48:03.094466 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 331) took 1305 ms. Trace:
I20251024 19:48:03.095031 1776 rpcz_store.cc:276] 1024 19:48:01.785950 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.793715 (+ 7765us) service_pool.cc:225] Handling call
1024 19:48:03.090693 (+1296978us) ts_tablet_manager.cc:1151] Deleting tablet 62667092d7e244cca1020ec220ea9124
1024 19:48:03.090703 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.091511 (+ 808us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1288781}
I20251024 19:48:03.095439 1776 tablet_replica.cc:333] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:03.095659 1776 raft_consensus.cc:2243] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:03.096072 1776 pending_rounds.cc:70] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:03.096185 1776 pending_rounds.cc:77] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429318314274816 op_type: NO_OP noop_request { }
I20251024 19:48:03.096304 1776 raft_consensus.cc:2889] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:03.096407 1776 raft_consensus.cc:2272] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:03.097210 1766 tablet_bootstrap.cc:654] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.098078 1776 ts_tablet_manager.cc:1916] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:03.098556 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 62667092d7e244cca1020ec220ea9124 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62667092d7e244cca1020ec220ea9124 already in progress: creating tablet
I20251024 19:48:03.053086 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 369cebfda6ee43638b79f8652fa55855 (table default.Table_c3cbf88afafa464099c69d8e017743e8 [id=27d44c1a6a2046de8d4e634e4eb6eb7e]) successfully deleted
W20251024 19:48:03.104467 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 51327.4us
I20251024 19:48:03.110780 1766 tablet_bootstrap.cc:492] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:03.110957 1766 ts_tablet_manager.cc:1403] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.000s
I20251024 19:48:03.111797 1766 raft_consensus.cc:359] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.111991 1766 raft_consensus.cc:385] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.112079 1766 raft_consensus.cc:740] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.112319 1766 consensus_queue.cc:260] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.112865 1766 ts_tablet_manager.cc:1434] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:03.113196 1766 tablet_bootstrap.cc:492] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:03.114081 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b6fb408e15354fb7b55bf182bb981be3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.114873 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6fb408e15354fb7b55bf182bb981be3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.116991 1766 tablet_bootstrap.cc:654] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.123560 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 92d78642772d44179e67d75c51b9da4e (DEFAULT_TABLE table=default.Table_3cd13b4df8f54c22aad4df0a6ccaef3e [id=37332f920f4d4827b16f1c1c7bffda6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.124418 1491 tablet_service.cc:1505] Processing CreateTablet for tablet ebcc09d8e029451d8fb6b6601ad16087 (DEFAULT_TABLE table=default.Table_546f0aaae7694ef9807723a9ecc39602 [id=3e8b487a98c64acd8c6c6dbb35197598]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.124305 1484 tablet_service.cc:1505] Processing CreateTablet for tablet b22b58404b8b441ba97ef536885a66e0 (DEFAULT_TABLE table=default.Table_f4e511aa3c4b46479c523daa4149707f [id=bdfe9c0f34ac41eebb843bd17fa9635c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.134510 1766 tablet_bootstrap.cc:492] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:03.134809 1766 ts_tablet_manager.cc:1403] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.003s
I20251024 19:48:03.135447 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7d1933bb728845abad5e6b09b24868f0
I20251024 19:48:03.135988 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9abaf6cefac3462491207156404d36b7" } new_table_name: "default.Table_f5d35eb71b3948ffbc2a9c015823e67f" modify_external_catalogs: true
W20251024 19:48:03.136612 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 505e4fb0755a4caaa462e5f5154bfee2 consensus state has no leader
I20251024 19:48:03.137163 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet d840888183ec4ef7bb245363c5bac322
I20251024 19:48:03.137518 1766 raft_consensus.cc:359] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.137786 1766 raft_consensus.cc:385] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.137873 1766 raft_consensus.cc:740] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.137923 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9a1031d785b646b095ece9be666d2a77
I20251024 19:48:03.138123 1766 consensus_queue.cc:260] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.138722 1766 ts_tablet_manager.cc:1434] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:03.139032 1766 tablet_bootstrap.cc:492] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:03.140792 1766 tablet_bootstrap.cc:654] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.142625 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a1031d785b646b095ece9be666d2a77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.146873 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet bc31e6c25ad14947af20abd766f12e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.147115 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet dc14b0cdb7f342ac956a7857ab3f35d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.147284 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet d76410afd7ba4c73836f4791ef739056 with delete_type TABLET_DATA_DELETED (replaced by tablet b153f75caebb4d869e77529c9c9bb50d) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.147645 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 3e6f8764e9ae472b9ac35a032182aed1 (DEFAULT_TABLE table=default.Table_aa7c283553a841e9ab2fa21fb6485a11 [id=7ec9391c52b1424eb52850f2c6ba4779]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:03.148188 1447 consensus_peers.cc:597] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 369cebfda6ee43638b79f8652fa55855. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:03.148473 2020 consensus_queue.cc:1048] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:03.150995 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet d840888183ec4ef7bb245363c5bac322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.156106 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d1933bb728845abad5e6b09b24868f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.161460 1775 ts_tablet_manager.cc:1929] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:03.161643 1775 log.cc:1199] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/b2ea545b2cc14f589eda9419e55234bf
I20251024 19:48:03.163187 1775 ts_tablet_manager.cc:1950] T b2ea545b2cc14f589eda9419e55234bf P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:03.163699 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e6f8764e9ae472b9ac35a032182aed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.164675 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b22b58404b8b441ba97ef536885a66e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.165320 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebcc09d8e029451d8fb6b6601ad16087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.165983 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92d78642772d44179e67d75c51b9da4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.166992 1767 tablet_bootstrap.cc:654] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.202224 1768 tablet_bootstrap.cc:492] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:03.202417 1768 ts_tablet_manager.cc:1403] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.137s user 0.013s sys 0.025s
I20251024 19:48:03.203294 1768 raft_consensus.cc:359] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.203501 1768 raft_consensus.cc:385] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.203626 1768 raft_consensus.cc:740] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.203923 1768 consensus_queue.cc:260] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.204593 1768 ts_tablet_manager.cc:1434] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:03.206532 1767 tablet_bootstrap.cc:492] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:03.206779 1767 ts_tablet_manager.cc:1403] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.158s user 0.002s sys 0.006s
I20251024 19:48:03.208544 1767 raft_consensus.cc:359] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.208741 1767 raft_consensus.cc:385] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.208837 1767 raft_consensus.cc:740] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.209070 1767 consensus_queue.cc:260] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.210299 1768 tablet_bootstrap.cc:492] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:03.212522 1768 tablet_bootstrap.cc:654] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.214743 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 0797d14e9fd846788164747523df0e2c (DEFAULT_TABLE table=default.Table_964aae2f085749baaf4f7d0ee3ebb1f4 [id=a07755e138c84fd18c0fe72e81ab758b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.215476 1202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef8906492b1a4368936fac95c11e1f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 335 } } ignore_live_leader: false dest_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" is_pre_election: true
I20251024 19:48:03.216594 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 31647a48fb834c80b6920c756a94250d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.217135 1767 ts_tablet_manager.cc:1434] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20251024 19:48:03.217118 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d1933bb728845abad5e6b09b24868f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.218149 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet d840888183ec4ef7bb245363c5bac322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.218483 1768 tablet_bootstrap.cc:492] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:03.218727 1768 ts_tablet_manager.cc:1403] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20251024 19:48:03.219012 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a1031d785b646b095ece9be666d2a77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.219316 2048 raft_consensus.cc:493] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.219470 2048 raft_consensus.cc:515] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.219619 1768 raft_consensus.cc:359] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.219800 1768 raft_consensus.cc:385] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.219893 1768 raft_consensus.cc:740] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.220175 2048 leader_election.cc:290] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:03.220726 1333 tablet_service.cc:1505] Processing CreateTablet for tablet bc31e6c25ad14947af20abd766f12e78 (DEFAULT_TABLE table=default.Table_2919f656bb2c466ab7679bc4a465c7ea [id=e0487ce7599645a2914ca3c8c5238b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.221336 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "897db285f5c84930a5b438be1cb4eb96" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:03.221907 1371 raft_consensus.cc:2468] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:03.222297 1586 leader_election.cc:304] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:03.222595 2158 raft_consensus.cc:493] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.222755 2158 raft_consensus.cc:515] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.223119 1767 tablet_bootstrap.cc:492] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:03.223351 2158 leader_election.cc:290] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:03.223577 2048 raft_consensus.cc:2804] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:03.223711 2048 raft_consensus.cc:493] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:03.223803 2048 raft_consensus.cc:3060] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:03.226285 1767 tablet_bootstrap.cc:654] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.227275 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2496f103436047dea5dabb577651249f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:03.227998 1305 leader_election.cc:343] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.233115 1768 consensus_queue.cc:260] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.234294 1768 ts_tablet_manager.cc:1434] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
W20251024 19:48:03.235054 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f89a3bbeb9b4d57a49078729110d951 consensus state has no leader
I20251024 19:48:03.235100 1768 tablet_bootstrap.cc:492] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:03.236915 1192 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]). Expected version 1 got 0
I20251024 19:48:03.237371 1768 tablet_bootstrap.cc:654] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.247995 1774 ts_tablet_manager.cc:1929] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:03.248193 1774 log.cc:1199] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/979dd87a8c66480ea314fcc75fa878c3
I20251024 19:48:03.248746 1774 ts_tablet_manager.cc:1950] T 979dd87a8c66480ea314fcc75fa878c3 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:03.253937 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0797d14e9fd846788164747523df0e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.259029 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "897db285f5c84930a5b438be1cb4eb96" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:03.259377 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc31e6c25ad14947af20abd766f12e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.259351 1508 raft_consensus.cc:2468] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:03.261205 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2496f103436047dea5dabb577651249f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:03.262218 1308 leader_election.cc:343] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.262392 2048 raft_consensus.cc:515] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.263038 2048 leader_election.cc:290] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:03.263103 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 309) took 1696 ms. Trace:
I20251024 19:48:03.263264 1472 tablet_service.cc:1552] Processing DeleteTablet for tablet d840888183ec4ef7bb245363c5bac322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.263342 1775 rpcz_store.cc:276] 1024 19:48:01.566241 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.574053 (+ 7812us) service_pool.cc:225] Handling call
1024 19:48:02.430902 (+856849us) ts_tablet_manager.cc:1151] Deleting tablet b2ea545b2cc14f589eda9419e55234bf
1024 19:48:02.430912 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.149653 (+718741us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.161299 (+ 11646us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.262650 (+101351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":856491}
I20251024 19:48:03.263824 1775 tablet_replica.cc:333] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:03.263877 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "897db285f5c84930a5b438be1cb4eb96" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:03.264055 1775 raft_consensus.cc:2243] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:03.264124 1508 raft_consensus.cc:3060] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:03.264223 1775 raft_consensus.cc:2272] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:03.265026 1775 ts_tablet_manager.cc:1916] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:03.265728 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a1031d785b646b095ece9be666d2a77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.267472 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d1933bb728845abad5e6b09b24868f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.268317 1308 leader_election.cc:304] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:03.268957 2158 raft_consensus.cc:2749] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:03.269234 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "897db285f5c84930a5b438be1cb4eb96" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:03.269423 1371 raft_consensus.cc:3060] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:03.269902 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet b2ea545b2cc14f589eda9419e55234bf (table default.Table_74349eed6d4b4e8e95ae01909bbb91e9 [id=94b8daec91b6478f9dc43734542246e0]) successfully deleted
I20251024 19:48:03.270298 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet ee3c521876324808857f7724028c895f from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:03.270874 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7e15b1eebd3443548f58b749626a8679"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.271473 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e15b1eebd3443548f58b749626a8679 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.272552 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 252) took 2923 ms. Trace:
I20251024 19:48:03.272704 1774 rpcz_store.cc:276] 1024 19:48:00.338266 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.338311 (+ 45us) service_pool.cc:225] Handling call
1024 19:48:03.016751 (+2678440us) ts_tablet_manager.cc:1151] Deleting tablet 979dd87a8c66480ea314fcc75fa878c3
1024 19:48:03.016764 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.152911 (+136147us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.247815 (+ 94904us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.261834 (+ 14019us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2648178}
I20251024 19:48:03.273270 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 255) took 2640 ms. Trace:
I20251024 19:48:03.273408 1774 rpcz_store.cc:276] 1024 19:48:00.632358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.632434 (+ 76us) service_pool.cc:225] Handling call
1024 19:48:03.273084 (+2640650us) ts_tablet_manager.cc:1151] Deleting tablet 7352d84226db467dad31ae5941842b80
1024 19:48:03.273095 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.273216 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2640358}
I20251024 19:48:03.273651 1916 leader_election.cc:304] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef8906492b1a4368936fac95c11e1f73; no voters: 82e82847f8904b59b03f446b7fce6233, dc8f71a5b70749a4a3e3a599bcb3d22c
I20251024 19:48:03.273886 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 256) took 2636 ms. Trace:
I20251024 19:48:03.273967 1774 rpcz_store.cc:276] 1024 19:48:00.637371 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.637425 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:03.273741 (+2636316us) ts_tablet_manager.cc:1151] Deleting tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f
1024 19:48:03.273751 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.273835 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2636040}
I20251024 19:48:03.274039 2174 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:48:03.274314 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 257) took 2633 ms. Trace:
I20251024 19:48:03.274410 1774 rpcz_store.cc:276] 1024 19:48:00.640409 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.640459 (+ 50us) service_pool.cc:225] Handling call
1024 19:48:03.274168 (+2633709us) ts_tablet_manager.cc:1151] Deleting tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7
1024 19:48:03.274179 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.274264 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2633517}
I20251024 19:48:03.276981 2133 catalog_manager.cc:1269] Loaded cluster ID: 57054fc705ad43a6b8ba5fe9c17ff20a
I20251024 19:48:03.277112 2133 catalog_manager.cc:1562] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: loading cluster ID for follower catalog manager: success
I20251024 19:48:03.277635 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 258) took 2634 ms. Trace:
I20251024 19:48:03.277734 1774 rpcz_store.cc:276] 1024 19:48:00.643404 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.643453 (+ 49us) service_pool.cc:225] Handling call
1024 19:48:03.274645 (+2631192us) ts_tablet_manager.cc:1151] Deleting tablet cef80e4d14bb4e719bceebf43041d079
1024 19:48:03.274654 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.277559 (+ 2905us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2630935}
I20251024 19:48:03.278178 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 0797d14e9fd846788164747523df0e2c (DEFAULT_TABLE table=default.Table_964aae2f085749baaf4f7d0ee3ebb1f4 [id=a07755e138c84fd18c0fe72e81ab758b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.278831 1491 tablet_service.cc:1505] Processing CreateTablet for tablet bc31e6c25ad14947af20abd766f12e78 (DEFAULT_TABLE table=default.Table_2919f656bb2c466ab7679bc4a465c7ea [id=e0487ce7599645a2914ca3c8c5238b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.287302 1768 tablet_bootstrap.cc:492] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:03.287489 1768 ts_tablet_manager.cc:1403] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.006s
I20251024 19:48:03.289855 1768 raft_consensus.cc:359] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.290050 1768 raft_consensus.cc:385] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.290135 1768 raft_consensus.cc:740] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.291584 1768 consensus_queue.cc:260] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.292577 1768 ts_tablet_manager.cc:1434] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:48:03.294665 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 897db285f5c84930a5b438be1cb4eb96 with delete_type TABLET_DATA_DELETED (replaced by tablet 758df87acbdb4adaa83e1ecbc44e7cf1) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.294921 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 31647a48fb834c80b6920c756a94250d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.298439 1511 raft_consensus.cc:1275] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Refusing update from remote peer 43bec2f03fc849099e254f6335e9d532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251024 19:48:03.298569 1305 consensus_peers.cc:597] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:03.299189 2130 consensus_queue.cc:1048] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:03.302237 1774 tablet_replica.cc:333] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:03.302531 1774 raft_consensus.cc:2243] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:03.302788 1774 raft_consensus.cc:2272] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:03.303478 1774 ts_tablet_manager.cc:1916] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:03.306200 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_34772bf3583943e4a3ffa064e9dd1333"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.307158 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34772bf3583943e4a3ffa064e9dd1333 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.310782 1768 tablet_bootstrap.cc:492] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:03.313884 1766 tablet_bootstrap.cc:492] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:03.314097 1766 ts_tablet_manager.cc:1403] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.175s user 0.024s sys 0.018s
I20251024 19:48:03.315621 1766 raft_consensus.cc:359] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:03.315881 1766 raft_consensus.cc:385] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.316076 1766 raft_consensus.cc:740] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.317874 1768 tablet_bootstrap.cc:654] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.319995 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_740023c3917c4850b564d096c64670d4" } new_table_name: "default.Table_bf15d3c700774490b1860d3d8865a95e" modify_external_catalogs: true
I20251024 19:48:03.321919 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_03318e0d18d34db48da05226fd40029a" } modify_external_catalogs: true
I20251024 19:48:03.322146 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_03318e0d18d34db48da05226fd40029a" } modify_external_catalogs: true
I20251024 19:48:03.322160 1766 consensus_queue.cc:260] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:03.323171 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_605117e2eb734e6b8dc34ceb4b5b5b71" } modify_external_catalogs: true
I20251024 19:48:03.323360 1768 tablet_bootstrap.cc:492] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:03.323376 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_605117e2eb734e6b8dc34ceb4b5b5b71" } modify_external_catalogs: true
I20251024 19:48:03.323511 1768 ts_tablet_manager.cc:1403] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20251024 19:48:03.324205 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5d986b803bac48beb19d3de7faf77cdf" } new_table_name: "default.Table_2d326fe7fd7347d4915d830eedc291a8" modify_external_catalogs: true
I20251024 19:48:03.324571 1766 ts_tablet_manager.cc:1434] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20251024 19:48:03.324898 1766 tablet_bootstrap.cc:492] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:03.327855 2133 catalog_manager.cc:1584] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: acquiring CA information for follower catalog manager: success
W20251024 19:48:03.335260 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d01572e51694d9bb58c3a59d4205102 consensus state has no leader
W20251024 19:48:03.340206 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 505e4fb0755a4caaa462e5f5154bfee2 consensus state has no leader
I20251024 19:48:03.343339 1768 raft_consensus.cc:359] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.343582 1768 raft_consensus.cc:385] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.343694 1768 raft_consensus.cc:740] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.343964 1768 consensus_queue.cc:260] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.344486 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f27b327a6f134fcbb33611d8bfcfc697" } new_table_name: "default.Table_d95afcc3eb2e481182edd9cffc3b5b50" modify_external_catalogs: true
I20251024 19:48:03.346551 1766 tablet_bootstrap.cc:654] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.346994 1371 raft_consensus.cc:2468] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:03.347016 1508 raft_consensus.cc:2468] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:03.347170 1776 ts_tablet_manager.cc:1929] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:03.347322 1776 log.cc:1199] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/b9b1e0338fd14634a1e921735ae61289
I20251024 19:48:03.347843 1776 ts_tablet_manager.cc:1950] T b9b1e0338fd14634a1e921735ae61289 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:03.348026 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0797d14e9fd846788164747523df0e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.348690 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc31e6c25ad14947af20abd766f12e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.351145 1768 ts_tablet_manager.cc:1434] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.028s user 0.000s sys 0.003s
I20251024 19:48:03.351938 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 335) took 1527 ms. Trace:
I20251024 19:48:03.352030 1768 tablet_bootstrap.cc:492] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:03.352077 1776 rpcz_store.cc:276] 1024 19:48:01.824704 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.824766 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:03.095304 (+1270538us) ts_tablet_manager.cc:1151] Deleting tablet b9b1e0338fd14634a1e921735ae61289
1024 19:48:03.095313 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.262023 (+166710us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.346963 (+ 84940us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.351894 (+ 4931us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1270118}
I20251024 19:48:03.352456 1776 tablet_replica.cc:333] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:03.352696 1776 raft_consensus.cc:2243] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:03.352902 1776 raft_consensus.cc:2272] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:03.353379 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet b9b1e0338fd14634a1e921735ae61289 (table default.Table_f02338f14f2b44b3b89757b7edc59e32 [id=3e4514469cd64a8b86d9e44608c186ba]) successfully deleted
I20251024 19:48:03.353857 1586 leader_election.cc:304] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:03.354158 2048 raft_consensus.cc:2804] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:03.354305 2048 raft_consensus.cc:697] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:03.354656 2048 consensus_queue.cc:237] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.356261 1776 ts_tablet_manager.cc:1916] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:03.357175 1768 tablet_bootstrap.cc:654] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.375520 2190 tablet.cc:1722] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251024 19:48:03.385275 1766 tablet_bootstrap.cc:492] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:03.385507 1766 ts_tablet_manager.cc:1403] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.007s
I20251024 19:48:03.386384 1766 raft_consensus.cc:359] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:03.386605 1766 raft_consensus.cc:385] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.386708 1766 raft_consensus.cc:740] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.389921 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0813796576a2420e8adafd30070bc747"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:03.391623 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5c266c8c172d4d08847b98402d5d0ffb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:03.387012 1766 consensus_queue.cc:260] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:03.394407 1766 ts_tablet_manager.cc:1434] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251024 19:48:03.395219 1766 tablet_bootstrap.cc:492] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:03.398089 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0813796576a2420e8adafd30070bc747 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:03.400158 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c266c8c172d4d08847b98402d5d0ffb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.400226 1766 tablet_bootstrap.cc:654] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.404973 1766 tablet_bootstrap.cc:492] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:03.405208 1766 ts_tablet_manager.cc:1403] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20251024 19:48:03.406687 1766 raft_consensus.cc:359] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.407027 1766 raft_consensus.cc:385] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.407269 1766 raft_consensus.cc:740] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.410825 1766 consensus_queue.cc:260] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.414103 1766 ts_tablet_manager.cc:1434] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.009s user 0.002s sys 0.002s
I20251024 19:48:03.414552 1766 tablet_bootstrap.cc:492] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:03.416435 1766 tablet_bootstrap.cc:654] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.419394 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet 701c93a600964c20a275f5942a2ff863 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:03.423243 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 979dd87a8c66480ea314fcc75fa878c3 (table default.Table_e0d4700f7fd849399f95eb388d194091 [id=cf70cd6b11614a0d8b39b707311c46fd]) successfully deleted
W20251024 19:48:03.423768 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7352d84226db467dad31ae5941842b80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7352d84226db467dad31ae5941842b80 already in progress: creating tablet
W20251024 19:48:03.424351 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 already in progress: creating tablet
I20251024 19:48:03.429984 1766 tablet_bootstrap.cc:492] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:03.430291 1766 ts_tablet_manager.cc:1403] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.005s
I20251024 19:48:03.431429 1766 raft_consensus.cc:359] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.431666 1766 raft_consensus.cc:385] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.431754 1766 raft_consensus.cc:740] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.431915 1767 tablet_bootstrap.cc:492] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:03.432189 1767 ts_tablet_manager.cc:1403] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.209s user 0.013s sys 0.029s
I20251024 19:48:03.434124 1766 consensus_queue.cc:260] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.434527 1767 raft_consensus.cc:359] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.434751 1767 raft_consensus.cc:385] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.434837 1767 raft_consensus.cc:740] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.435089 1767 consensus_queue.cc:260] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.436651 1766 ts_tablet_manager.cc:1434] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20251024 19:48:03.436952 1766 tablet_bootstrap.cc:492] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:03.438685 2048 consensus_queue.cc:1048] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:03.438939 1766 tablet_bootstrap.cc:654] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.440712 1767 ts_tablet_manager.cc:1434] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20251024 19:48:03.441661 2020 raft_consensus.cc:493] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.441804 2020 raft_consensus.cc:515] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:03.442098 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 9623d7baf0174a019e2353a5effef081 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.442456 2020 leader_election.cc:290] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:03.443308 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31647a48fb834c80b6920c756a94250d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:03.443534 1648 raft_consensus.cc:2468] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
W20251024 19:48:03.444252 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet cef80e4d14bb4e719bceebf43041d079 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cef80e4d14bb4e719bceebf43041d079 already in progress: creating tablet
I20251024 19:48:03.444416 1444 leader_election.cc:304] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:03.431234 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 452f77091adc4cf09de2231f2f6d845b consensus state has no leader
I20251024 19:48:03.431624 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d7a9ef5436984a1b83dbeac77c302549" } modify_external_catalogs: true
I20251024 19:48:03.445055 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d7a9ef5436984a1b83dbeac77c302549" } modify_external_catalogs: true
I20251024 19:48:03.445220 2020 raft_consensus.cc:2804] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:03.445348 2020 raft_consensus.cc:493] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:03.432026 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0dfdcd7f531e4ddca00916bf1dcec977"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:03.445432 2020 raft_consensus.cc:3060] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:03.445936 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dfdcd7f531e4ddca00916bf1dcec977 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.437031 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_740632e4f743422b850972065ef1f415" } new_table_name: "default.Table_c13d19a355a24f2598c8d5741bbe8521" modify_external_catalogs: true
I20251024 19:48:03.444741 2189 tablet.cc:1722] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251024 19:48:03.450266 1775 ts_tablet_manager.cc:1929] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:03.450428 1775 log.cc:1199] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/979dd87a8c66480ea314fcc75fa878c3
I20251024 19:48:03.450914 1775 ts_tablet_manager.cc:1950] T 979dd87a8c66480ea314fcc75fa878c3 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:03.453725 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31647a48fb834c80b6920c756a94250d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:03.454021 1371 raft_consensus.cc:2468] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:03.455478 1767 tablet_bootstrap.cc:492] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:03.457317 1767 tablet_bootstrap.cc:654] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.458474 2020 raft_consensus.cc:515] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:03.459856 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31647a48fb834c80b6920c756a94250d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:03.460093 1648 raft_consensus.cc:3060] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:03.463083 2020 leader_election.cc:290] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:03.465287 1776 ts_tablet_manager.cc:1929] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:03.465539 1776 log.cc:1199] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/369cebfda6ee43638b79f8652fa55855
I20251024 19:48:03.466167 1776 ts_tablet_manager.cc:1950] T 369cebfda6ee43638b79f8652fa55855 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:03.468593 1766 tablet_bootstrap.cc:492] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:03.468801 1766 ts_tablet_manager.cc:1403] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.001s
I20251024 19:48:03.468770 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31647a48fb834c80b6920c756a94250d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:03.468994 1371 raft_consensus.cc:3060] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:03.469653 1766 raft_consensus.cc:359] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.469857 1766 raft_consensus.cc:385] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.469946 1766 raft_consensus.cc:740] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.470163 1766 consensus_queue.cc:260] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.471805 1766 ts_tablet_manager.cc:1434] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:03.473462 1192 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]). Expected version 1 got 0
I20251024 19:48:03.478987 1768 tablet_bootstrap.cc:492] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:03.479509 1768 ts_tablet_manager.cc:1403] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.127s user 0.016s sys 0.018s
I20251024 19:48:03.480310 1768 raft_consensus.cc:359] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.480535 1768 raft_consensus.cc:385] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.480615 1768 raft_consensus.cc:740] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.480865 1768 consensus_queue.cc:260] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.481379 1768 ts_tablet_manager.cc:1434] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:03.482952 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 8d1b6eaa353c4284a2b114b42437272c (DEFAULT_TABLE table=default.Table_e25a14d3a9674ad38cce670621e477df [id=51f5913271754b6f86525a41d9a0ce22]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.483408 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d1b6eaa353c4284a2b114b42437272c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.484916 1768 tablet_bootstrap.cc:492] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:03.487493 2154 raft_consensus.cc:493] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.487653 2154 raft_consensus.cc:515] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.488286 2154 leader_election.cc:290] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:03.489235 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a7e76a3cac443ec8d2ab466075aae6b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:03.489742 1308 leader_election.cc:343] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.490418 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a7e76a3cac443ec8d2ab466075aae6b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:03.491384 1305 leader_election.cc:343] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.491525 1305 leader_election.cc:304] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:03.494086 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 2671108031db449cae2135835fc5f739 (DEFAULT_TABLE table=default.Table_d5aeb6bab67b4c1ea34aa869731de12a [id=2f6b7b685af84337bd64141ef2a9e992]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.494129 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_34ef66bee6bd460b92cf30b469b3f179" } modify_external_catalogs: true
I20251024 19:48:03.494535 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2671108031db449cae2135835fc5f739. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.494480 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_34ef66bee6bd460b92cf30b469b3f179" } modify_external_catalogs: true
W20251024 19:48:03.494568 1586 consensus_peers.cc:597] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c292ce055def4964ab143252c6b2b87e. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:03.494881 1768 tablet_bootstrap.cc:654] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.495008 2154 raft_consensus.cc:2749] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:03.496187 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b8e34cb629054ed9be6deaf4d30992a2" } modify_external_catalogs: true
I20251024 19:48:03.496418 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b8e34cb629054ed9be6deaf4d30992a2" } modify_external_catalogs: true
W20251024 19:48:03.502804 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee3c521876324808857f7724028c895f consensus state has no leader
I20251024 19:48:03.504546 1768 tablet_bootstrap.cc:492] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:03.504745 1768 ts_tablet_manager.cc:1403] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20251024 19:48:03.506127 1768 raft_consensus.cc:359] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.506444 1768 raft_consensus.cc:385] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.506670 1768 raft_consensus.cc:740] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.507073 1768 consensus_queue.cc:260] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.507910 1768 ts_tablet_manager.cc:1434] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:03.508266 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 38818c7997c84cedad5f9b09eef77105
I20251024 19:48:03.508415 1768 tablet_bootstrap.cc:492] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:03.509785 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 38818c7997c84cedad5f9b09eef77105 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.513885 1768 tablet_bootstrap.cc:654] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.517776 1766 tablet_bootstrap.cc:492] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:03.519425 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet 2602335327fc489fa154056c5ef5138f from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:03.519779 1628 tablet_service.cc:1505] Processing CreateTablet for tablet aa08555dbe1e4ad8968c534510eca9ff (DEFAULT_TABLE table=default.Table_ba89fbfa74a0425c9db8f48b5f94eb83 [id=79c88afabb0d4729a959acc2c39ef387]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.520256 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa08555dbe1e4ad8968c534510eca9ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.520273 1766 tablet_bootstrap.cc:654] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.520639 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 8d1b6eaa353c4284a2b114b42437272c (DEFAULT_TABLE table=default.Table_e25a14d3a9674ad38cce670621e477df [id=51f5913271754b6f86525a41d9a0ce22]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.521886 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet 38818c7997c84cedad5f9b09eef77105 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.522516 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 8d1b6eaa353c4284a2b114b42437272c (DEFAULT_TABLE table=default.Table_e25a14d3a9674ad38cce670621e477df [id=51f5913271754b6f86525a41d9a0ce22]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.522553 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a2f9ff12c83949f89a0a0442e8f8189e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.523381 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2f9ff12c83949f89a0a0442e8f8189e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.527252 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b5a5283ea815484a91f644c13a1826b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.528097 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5a5283ea815484a91f644c13a1826b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:03.529906 1177 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet ee3c521876324808857f7724028c895f (table default.Table_c13d19a355a24f2598c8d5741bbe8521 [id=f49bfd0c428d44f69a3d97a6c51a380a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ee3c521876324808857f7724028c895f consensus state has no leader
I20251024 19:48:03.535660 2020 raft_consensus.cc:493] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.535849 2020 raft_consensus.cc:515] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.536564 2020 leader_election.cc:290] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:03.537308 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2496f103436047dea5dabb577651249f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:03.539812 1444 leader_election.cc:343] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.540853 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2496f103436047dea5dabb577651249f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:03.541168 1372 raft_consensus.cc:2468] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:03.541755 1447 leader_election.cc:304] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:03.542263 2020 raft_consensus.cc:2804] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:03.542384 2020 raft_consensus.cc:493] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:03.542490 2020 raft_consensus.cc:3060] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:03.544090 1648 raft_consensus.cc:2468] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:03.545883 1371 raft_consensus.cc:2468] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:03.548192 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 317) took 1953 ms. Trace:
I20251024 19:48:03.548319 1775 rpcz_store.cc:276] 1024 19:48:01.594362 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.594428 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:03.263754 (+1669326us) ts_tablet_manager.cc:1151] Deleting tablet 979dd87a8c66480ea314fcc75fa878c3
1024 19:48:03.263765 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.347430 (+ 83665us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.450140 (+102710us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.548114 (+ 97974us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1669110}
I20251024 19:48:03.548993 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 318) took 1898 ms. Trace:
I20251024 19:48:03.549108 1775 rpcz_store.cc:276] 1024 19:48:01.650219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.656318 (+ 6099us) service_pool.cc:225] Handling call
1024 19:48:03.548676 (+1892358us) ts_tablet_manager.cc:1151] Deleting tablet 758df87acbdb4adaa83e1ecbc44e7cf1
1024 19:48:03.548689 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.548813 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1891999}
I20251024 19:48:03.549538 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 319) took 1883 ms. Trace:
I20251024 19:48:03.551515 1775 rpcz_store.cc:276] 1024 19:48:01.666211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.667525 (+ 1314us) service_pool.cc:225] Handling call
1024 19:48:03.549367 (+1881842us) ts_tablet_manager.cc:1151] Deleting tablet dc1b6f0a377945089c50012f74d38564
1024 19:48:03.549379 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.549472 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1881554}
I20251024 19:48:03.551913 1775 tablet_replica.cc:333] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:03.552075 1444 leader_election.cc:304] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:03.552181 1775 raft_consensus.cc:2243] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:03.552392 2141 raft_consensus.cc:2804] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:03.552556 2141 raft_consensus.cc:697] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:03.552882 2141 consensus_queue.cc:237] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:03.555781 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 2671108031db449cae2135835fc5f739 (DEFAULT_TABLE table=default.Table_d5aeb6bab67b4c1ea34aa869731de12a [id=2f6b7b685af84337bd64141ef2a9e992]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.556350 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2671108031db449cae2135835fc5f739. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.557509 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 2671108031db449cae2135835fc5f739 (DEFAULT_TABLE table=default.Table_d5aeb6bab67b4c1ea34aa869731de12a [id=2f6b7b685af84337bd64141ef2a9e992]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.560837 1766 tablet_bootstrap.cc:492] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:03.561024 1766 ts_tablet_manager.cc:1403] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.003s
I20251024 19:48:03.561618 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 336) took 1685 ms. Trace:
I20251024 19:48:03.561794 1776 rpcz_store.cc:276] 1024 19:48:01.865280 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.865334 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:03.352385 (+1487051us) ts_tablet_manager.cc:1151] Deleting tablet 369cebfda6ee43638b79f8652fa55855
1024 19:48:03.352398 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.449639 (+ 97241us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.465118 (+ 15479us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.551046 (+ 85928us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1445967}
I20251024 19:48:03.562019 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 701c93a600964c20a275f5942a2ff863
I20251024 19:48:03.562490 1191 catalog_manager.cc:6719] ReplaceTablet: tablet 701c93a600964c20a275f5942a2ff863 deleted and replaced by tablet 7da1601a87524ef1a8b5069ba6823843
I20251024 19:48:03.562389 1766 raft_consensus.cc:359] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.562801 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d1b6eaa353c4284a2b114b42437272c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.562849 1766 raft_consensus.cc:385] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.562930 1766 raft_consensus.cc:740] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.563251 1766 consensus_queue.cc:260] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.563913 1766 ts_tablet_manager.cc:1434] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:03.564325 1766 tablet_bootstrap.cc:492] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:03.565855 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 786fe2fd6fca407e83a76f74f96a8b16
I20251024 19:48:03.567662 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet c04082bd1b444b468c177bef239701c6 (table default.Table_24200cd6f19d460cae963afb9b7f8724 [id=a2789faf4abe4c09b0a7515d941831a6]). Expected version 2 got 0
I20251024 19:48:03.567704 1775 raft_consensus.cc:2272] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:03.568377 1775 ts_tablet_manager.cc:1916] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:03.572422 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 369cebfda6ee43638b79f8652fa55855 (table default.Table_c3cbf88afafa464099c69d8e017743e8 [id=27d44c1a6a2046de8d4e634e4eb6eb7e]) successfully deleted
I20251024 19:48:03.575145 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 38818c7997c84cedad5f9b09eef77105 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.575639 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 344) took 1587 ms. Trace:
I20251024 19:48:03.575757 1776 rpcz_store.cc:276] 1024 19:48:01.984241 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.997643 (+ 13402us) service_pool.cc:225] Handling call
1024 19:48:03.562115 (+1564472us) ts_tablet_manager.cc:1151] Deleting tablet 1edc55caab904722bf19a1c71df601cf
1024 19:48:03.571697 (+ 9582us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.571833 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1564100}
I20251024 19:48:03.576265 1766 tablet_bootstrap.cc:654] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.577297 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 345) took 1565 ms. Trace:
I20251024 19:48:03.577415 1776 rpcz_store.cc:276] 1024 19:48:02.010384 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.010536 (+ 152us) service_pool.cc:225] Handling call
1024 19:48:03.576016 (+1565480us) ts_tablet_manager.cc:1151] Deleting tablet e3b4e85f741849b2a6a3e8574396ff88
1024 19:48:03.576041 (+ 25us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.576150 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565062}
I20251024 19:48:03.579267 1776 tablet_replica.cc:333] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:03.579526 1776 raft_consensus.cc:2243] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:03.582036 1629 tablet_service.cc:1505] Processing CreateTablet for tablet b153f75caebb4d869e77529c9c9bb50d (DEFAULT_TABLE table=default.Table_34bbddd692a44f46a4925c957f4e8848 [id=6f9b0ed7552c4556afd0f59316831a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.582607 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b153f75caebb4d869e77529c9c9bb50d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.585152 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 6ef2add387da44868dbd2b5724ee7d86 (DEFAULT_TABLE table=default.Table_50d055bf7fc34e8fa68f1a188aea6218 [id=e55575a8212b40e093444ad77f9835e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.585896 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ef2add387da44868dbd2b5724ee7d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.586427 1346 tablet_service.cc:1505] Processing CreateTablet for tablet aa08555dbe1e4ad8968c534510eca9ff (DEFAULT_TABLE table=default.Table_ba89fbfa74a0425c9db8f48b5f94eb83 [id=79c88afabb0d4729a959acc2c39ef387]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.586651 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 701c93a600964c20a275f5942a2ff863 with delete_type TABLET_DATA_DELETED (replaced by tablet 7da1601a87524ef1a8b5069ba6823843) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:03.588011 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 1edc55caab904722bf19a1c71df601cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1edc55caab904722bf19a1c71df601cf already in progress: creating tablet
W20251024 19:48:03.588320 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet e3b4e85f741849b2a6a3e8574396ff88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3b4e85f741849b2a6a3e8574396ff88 already in progress: creating tablet
I20251024 19:48:03.589844 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 786fe2fd6fca407e83a76f74f96a8b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.589807 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 6ef2add387da44868dbd2b5724ee7d86 (DEFAULT_TABLE table=default.Table_50d055bf7fc34e8fa68f1a188aea6218 [id=e55575a8212b40e093444ad77f9835e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.590636 1344 tablet_service.cc:1505] Processing CreateTablet for tablet b153f75caebb4d869e77529c9c9bb50d (DEFAULT_TABLE table=default.Table_34bbddd692a44f46a4925c957f4e8848 [id=6f9b0ed7552c4556afd0f59316831a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.591709 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 701c93a600964c20a275f5942a2ff863 with delete_type TABLET_DATA_DELETED (replaced by tablet 7da1601a87524ef1a8b5069ba6823843) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.592828 2133 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251024 19:48:03.593416 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 786fe2fd6fca407e83a76f74f96a8b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.593933 1776 raft_consensus.cc:2272] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:03.594952 1776 ts_tablet_manager.cc:1916] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:03.601187 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 979dd87a8c66480ea314fcc75fa878c3 (table default.Table_e0d4700f7fd849399f95eb388d194091 [id=cf70cd6b11614a0d8b39b707311c46fd]) successfully deleted
W20251024 19:48:03.601516 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 758df87acbdb4adaa83e1ecbc44e7cf1 already in progress: creating tablet
I20251024 19:48:03.603078 1487 tablet_service.cc:1505] Processing CreateTablet for tablet aa08555dbe1e4ad8968c534510eca9ff (DEFAULT_TABLE table=default.Table_ba89fbfa74a0425c9db8f48b5f94eb83 [id=79c88afabb0d4729a959acc2c39ef387]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.603798 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 6ef2add387da44868dbd2b5724ee7d86 (DEFAULT_TABLE table=default.Table_50d055bf7fc34e8fa68f1a188aea6218 [id=e55575a8212b40e093444ad77f9835e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.611415 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa08555dbe1e4ad8968c534510eca9ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.612041 1472 tablet_service.cc:1505] Processing CreateTablet for tablet b153f75caebb4d869e77529c9c9bb50d (DEFAULT_TABLE table=default.Table_34bbddd692a44f46a4925c957f4e8848 [id=6f9b0ed7552c4556afd0f59316831a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.616600 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ef2add387da44868dbd2b5724ee7d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.616676 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet ee68243b4a664575b679950603bb250e from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:03.619019 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 786fe2fd6fca407e83a76f74f96a8b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.619295 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 701c93a600964c20a275f5942a2ff863 with delete_type TABLET_DATA_DELETED (replaced by tablet 7da1601a87524ef1a8b5069ba6823843) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.629710 1768 tablet_bootstrap.cc:492] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:03.629817 2154 raft_consensus.cc:493] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.629921 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ee3c521876324808857f7724028c895f
I20251024 19:48:03.630201 2154 raft_consensus.cc:515] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.630946 1186 catalog_manager.cc:6719] ReplaceTablet: tablet ee3c521876324808857f7724028c895f deleted and replaced by tablet 94b35252e63f4a8ba305cc029ddb5ae8
I20251024 19:48:03.631541 2154 leader_election.cc:290] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:03.632618 1768 ts_tablet_manager.cc:1403] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.122s user 0.019s sys 0.019s
I20251024 19:48:03.634325 1767 tablet_bootstrap.cc:492] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:03.634619 1767 ts_tablet_manager.cc:1403] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.179s user 0.014s sys 0.033s
I20251024 19:48:03.636178 1768 raft_consensus.cc:359] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.636729 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet ee3c521876324808857f7724028c895f with delete_type TABLET_DATA_DELETED (replaced by tablet 94b35252e63f4a8ba305cc029ddb5ae8) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.637090 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c357a2a81064b3891afda6d252898b9" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:03.638298 1767 raft_consensus.cc:359] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:03.638579 1767 raft_consensus.cc:385] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.638738 1767 raft_consensus.cc:740] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.638697 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_63c72c09261e44538e2cb5586550b43f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:03.639024 1767 consensus_queue.cc:260] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:03.639465 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63c72c09261e44538e2cb5586550b43f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.640923 1767 ts_tablet_manager.cc:1434] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251024 19:48:03.641230 1767 tablet_bootstrap.cc:492] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:03.643971 1767 tablet_bootstrap.cc:654] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.645056 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ee68243b4a664575b679950603bb250e
I20251024 19:48:03.646672 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet ee68243b4a664575b679950603bb250e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.637903 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet ee3c521876324808857f7724028c895f with delete_type TABLET_DATA_DELETED (replaced by tablet 94b35252e63f4a8ba305cc029ddb5ae8) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:03.638181 1305 leader_election.cc:343] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.652033 1768 raft_consensus.cc:385] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.652148 1768 raft_consensus.cc:740] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.649346 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d1b6eaa353c4284a2b114b42437272c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.652459 1768 consensus_queue.cc:260] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.655517 1768 ts_tablet_manager.cc:1434] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20251024 19:48:03.655885 1768 tablet_bootstrap.cc:492] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:03.657768 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa08555dbe1e4ad8968c534510eca9ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.657838 1768 tablet_bootstrap.cc:654] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.661754 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b153f75caebb4d869e77529c9c9bb50d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.662214 2154 raft_consensus.cc:493] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.662364 2154 raft_consensus.cc:515] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.663069 2154 leader_election.cc:290] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:03.664005 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913534ab88dd459e9cac2faebb5492a1" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:03.664544 1305 leader_election.cc:343] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.665649 1768 tablet_bootstrap.cc:492] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:03.665707 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ef2add387da44868dbd2b5724ee7d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.665802 1768 ts_tablet_manager.cc:1403] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20251024 19:48:03.669504 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2671108031db449cae2135835fc5f739. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.666620 1768 raft_consensus.cc:359] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.669936 1768 raft_consensus.cc:385] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.670047 1768 raft_consensus.cc:740] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.670647 1768 consensus_queue.cc:260] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.671355 2154 raft_consensus.cc:493] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.671633 2154 raft_consensus.cc:515] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.672531 2146 raft_consensus.cc:493] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.672827 2146 raft_consensus.cc:515] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.673800 2146 leader_election.cc:290] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:03.676523 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50d919a0b3d74e9a97d136a84d50e24b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:03.680151 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ba89fbfa74a0425c9db8f48b5f94eb83" } new_table_name: "default.Table_b33dc95780534701ab219c111a03ee38" modify_external_catalogs: true
I20251024 19:48:03.683624 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4c1febbedc3448728852a72f7f5d2d44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:03.686314 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet e20881e400cf4fac914b70afc79c084f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:03.686651 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c1febbedc3448728852a72f7f5d2d44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:03.689700 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 96182.9us
I20251024 19:48:03.695020 1189 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_03760ade20c7464f81dd428a3eb734fb [id=aadca5a1b6d4494aa37f8a8cc76ad12a] alter complete (version 5)
W20251024 19:48:03.695415 1305 leader_election.cc:343] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.695952 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c357a2a81064b3891afda6d252898b9" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:03.696038 2020 raft_consensus.cc:515] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.696416 2158 raft_consensus.cc:493] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.696585 2158 raft_consensus.cc:515] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.696964 2020 leader_election.cc:290] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:03.697187 2158 leader_election.cc:290] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:03.697253 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b153f75caebb4d869e77529c9c9bb50d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.697413 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet ee68243b4a664575b679950603bb250e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.699184 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2496f103436047dea5dabb577651249f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:03.700371 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c357a2a81064b3891afda6d252898b9 with delete_type TABLET_DATA_DELETED (replaced by tablet d38154d3ffd34f4b8d7305b1c23a4fae) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.700955 2020 raft_consensus.cc:493] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.701143 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d1933bb728845abad5e6b09b24868f0" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:03.701179 2020 raft_consensus.cc:515] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:03.702108 1305 leader_election.cc:343] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.702463 2154 leader_election.cc:290] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:03.702775 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913534ab88dd459e9cac2faebb5492a1" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:03.702407 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5834ffa88ad9439a9052f00f4859d799"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.703909 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5834ffa88ad9439a9052f00f4859d799 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:03.704414 1586 leader_election.cc:343] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 9623d7baf0174a019e2353a5effef081
I20251024 19:48:03.704331 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b9d913c84cec4d508db7fe4c3dc63e27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.706379 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9d913c84cec4d508db7fe4c3dc63e27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:03.706383 1308 leader_election.cc:343] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.706516 1308 leader_election.cc:304] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:03.707757 2154 raft_consensus.cc:2749] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:03.709535 2020 leader_election.cc:290] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:03.711884 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 346383e85cc64b3a8da70a71868ceccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.712144 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 897db285f5c84930a5b438be1cb4eb96 with delete_type TABLET_DATA_DELETED (replaced by tablet 758df87acbdb4adaa83e1ecbc44e7cf1) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.712284 1767 tablet_bootstrap.cc:492] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:03.712384 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet a48249f1824a40b087b80a563816d485 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.712457 1767 ts_tablet_manager.cc:1403] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.071s user 0.009s sys 0.008s
I20251024 19:48:03.712651 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet b5473699c1b647f8b86ee43fdb56bdfd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.713111 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet e20881e400cf4fac914b70afc79c084f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.713289 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 31647a48fb834c80b6920c756a94250d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.713356 1767 raft_consensus.cc:359] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.713557 1767 raft_consensus.cc:385] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.713663 1767 raft_consensus.cc:740] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.713940 1767 consensus_queue.cc:260] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.714012 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9623d7baf0174a019e2353a5effef081" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:03.714500 1767 ts_tablet_manager.cc:1434] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251024 19:48:03.715049 1767 tablet_bootstrap.cc:492] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:03.717106 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 45cff9bcc4924e0081518c284f074940
I20251024 19:48:03.717317 1767 tablet_bootstrap.cc:654] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.718631 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 45cff9bcc4924e0081518c284f074940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.721019 1508 raft_consensus.cc:2468] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:03.721328 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2496f103436047dea5dabb577651249f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:03.722594 1586 leader_election.cc:304] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:03.722889 2146 raft_consensus.cc:2804] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:03.723131 2146 raft_consensus.cc:493] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:03.723097 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7af095414d2e40c08193c763d497a755"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:03.723306 2146 raft_consensus.cc:3060] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:03.723771 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7af095414d2e40c08193c763d497a755 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.726339 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e20881e400cf4fac914b70afc79c084f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:03.726572 1648 raft_consensus.cc:2468] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:03.726938 1371 raft_consensus.cc:3060] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:03.728643 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet ee3c521876324808857f7724028c895f with delete_type TABLET_DATA_DELETED (replaced by tablet 94b35252e63f4a8ba305cc029ddb5ae8) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.731832 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 857eed60ba2a4b309e86d88a0f093762
I20251024 19:48:03.733367 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50d919a0b3d74e9a97d136a84d50e24b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:03.733937 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 857eed60ba2a4b309e86d88a0f093762 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:03.736243 1444 leader_election.cc:343] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.731431 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e20881e400cf4fac914b70afc79c084f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:03.740212 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 45cff9bcc4924e0081518c284f074940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:03.742311 1308 leader_election.cc:343] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.743803 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet c04082bd1b444b468c177bef239701c6 (table default.Table_24200cd6f19d460cae963afb9b7f8724 [id=a2789faf4abe4c09b0a7515d941831a6]). Expected version 2 got 0
I20251024 19:48:03.744778 2146 raft_consensus.cc:515] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.745448 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2d326fe7fd7347d4915d830eedc291a8" } new_table_name: "default.Table_939063dc1bea4e6d9d22b04cafce746b" modify_external_catalogs: true
I20251024 19:48:03.745900 2146 leader_election.cc:290] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:03.747877 1774 ts_tablet_manager.cc:1929] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:03.748013 1774 log.cc:1199] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/e20881e400cf4fac914b70afc79c084f
I20251024 19:48:03.748860 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 857eed60ba2a4b309e86d88a0f093762 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.749540 1308 leader_election.cc:304] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:03.750223 2154 raft_consensus.cc:2749] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:03.750233 1774 ts_tablet_manager.cc:1950] T e20881e400cf4fac914b70afc79c084f P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:03.751145 1333 tablet_service.cc:1552] Processing DeleteTablet for tablet 45cff9bcc4924e0081518c284f074940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.751631 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 259) took 3104 ms. Trace:
I20251024 19:48:03.751832 1776 ts_tablet_manager.cc:1929] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:03.751961 1776 log.cc:1199] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/c292ce055def4964ab143252c6b2b87e
I20251024 19:48:03.751907 1774 rpcz_store.cc:276] 1024 19:48:00.646864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.646916 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:03.277959 (+2631043us) ts_tablet_manager.cc:1151] Deleting tablet e20881e400cf4fac914b70afc79c084f
1024 19:48:03.302089 (+ 24130us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.648738 (+346649us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.747487 (+ 98749us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.750851 (+ 3364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2630841}
I20251024 19:48:03.752343 1774 tablet_replica.cc:333] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:03.752389 1776 ts_tablet_manager.cc:1950] T c292ce055def4964ab143252c6b2b87e P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:03.752955 1774 raft_consensus.cc:2243] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:03.753127 1774 raft_consensus.cc:2272] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:03.754454 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet e20881e400cf4fac914b70afc79c084f (table default.Table_69e288a9d6d24a5f841f62cd9a208f0f [id=969d428d45db42d18a5a9b82ec945bfb]) successfully deleted
I20251024 19:48:03.754590 1774 ts_tablet_manager.cc:1916] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:03.759752 2154 raft_consensus.cc:493] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.759837 1768 ts_tablet_manager.cc:1434] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.057s user 0.002s sys 0.000s
I20251024 19:48:03.759930 2154 raft_consensus.cc:515] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.760227 1768 tablet_bootstrap.cc:492] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:03.760646 2154 leader_election.cc:290] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:03.762071 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f346a90a9efe442f82750336e3893deb" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:03.762380 1768 tablet_bootstrap.cc:654] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.764848 1768 tablet_bootstrap.cc:492] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:03.765048 1768 ts_tablet_manager.cc:1403] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
W20251024 19:48:03.765914 1305 leader_election.cc:343] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.766422 1768 raft_consensus.cc:359] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.766851 1768 raft_consensus.cc:385] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.767007 1768 raft_consensus.cc:740] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.767730 1768 consensus_queue.cc:260] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.770347 1768 ts_tablet_manager.cc:1434] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251024 19:48:03.770833 1768 tablet_bootstrap.cc:492] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:03.773799 1768 tablet_bootstrap.cc:654] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:03.775310 1586 leader_election.cc:343] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 9623d7baf0174a019e2353a5effef081
I20251024 19:48:03.775647 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2602335327fc489fa154056c5ef5138f
I20251024 19:48:03.776310 1187 catalog_manager.cc:6719] ReplaceTablet: tablet 2602335327fc489fa154056c5ef5138f deleted and replaced by tablet 74029a7d7cdc4503a833c1758887b684
I20251024 19:48:03.778451 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 2602335327fc489fa154056c5ef5138f with delete_type TABLET_DATA_DELETED (replaced by tablet 74029a7d7cdc4503a833c1758887b684) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:03.778833 1333 tablet_service.cc:1552] Processing DeleteTablet for tablet 2602335327fc489fa154056c5ef5138f with delete_type TABLET_DATA_DELETED (replaced by tablet 74029a7d7cdc4503a833c1758887b684) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.782115 1333 tablet_service.cc:1552] Processing DeleteTablet for tablet 50d919a0b3d74e9a97d136a84d50e24b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.782310 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet f346a90a9efe442f82750336e3893deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.782392 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet b5473699c1b647f8b86ee43fdb56bdfd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.783409 1766 tablet_bootstrap.cc:492] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:03.783582 1766 ts_tablet_manager.cc:1403] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.219s user 0.024s sys 0.054s
I20251024 19:48:03.784489 1444 leader_election.cc:304] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:03.785041 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 857eed60ba2a4b309e86d88a0f093762 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.785113 1775 ts_tablet_manager.cc:1929] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:03.785315 1775 log.cc:1199] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/9820e68ad5264df1a79b22f3b5369c5a
I20251024 19:48:03.786119 1775 ts_tablet_manager.cc:1950] T 9820e68ad5264df1a79b22f3b5369c5a P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:03.787042 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9623d7baf0174a019e2353a5effef081" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:03.787312 1508 raft_consensus.cc:3060] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:03.788157 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa08555dbe1e4ad8968c534510eca9ff consensus state has no leader
W20251024 19:48:03.793043 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 505e4fb0755a4caaa462e5f5154bfee2 consensus state has no leader
I20251024 19:48:03.793409 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_61a8b015b11f4effa0245b9737c3c2ff" } modify_external_catalogs: true
I20251024 19:48:03.793591 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_61a8b015b11f4effa0245b9737c3c2ff" } modify_external_catalogs: true
I20251024 19:48:03.793502 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_edbcecddf69d4de8af2cf58ac7b12902"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:02.168558 1960 rpcz_store.cc:276] 1024 19:48:00.503632 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.503795 (+ 163us) service_pool.cc:225] Handling call
1024 19:48:00.769993 (+266198us) raft_consensus.cc:1471] Updating replica for 321 ops
1024 19:48:00.867137 (+ 97144us) raft_consensus.cc:1534] Early marking committed up to index 14
1024 19:48:00.880791 (+ 13654us) raft_consensus.cc:1539] Triggering prepare for 321 ops
1024 19:48:00.881295 (+ 504us) write_op.cc:270] Start()
1024 19:48:00.881355 (+ 60us) write_op.cc:276] Timestamp: P: 1761335275306572 usec, L: 0
1024 19:48:00.881774 (+ 419us) write_op.cc:270] Start()
1024 19:48:00.881829 (+ 55us) write_op.cc:276] Timestamp: P: 1761335275311331 usec, L: 0
1024 19:48:00.882049 (+ 220us) write_op.cc:270] Start()
1024 19:48:00.882082 (+ 33us) write_op.cc:276] Timestamp: P: 1761335275332970 usec, L: 0
1024 19:48:00.882239 (+ 157us) write_op.cc:270] Start()
1024 19:48:00.882261 (+ 22us) write_op.cc:276] Timestamp: P: 1761335275334123 usec, L: 0
1024 19:48:00.882444 (+ 183us) write_op.cc:270] Start()
1024 19:48:00.882469 (+ 25us) write_op.cc:276] Timestamp: P: 1761335275334720 usec, L: 0
1024 19:48:00.882657 (+ 188us) write_op.cc:270] Start()
1024 19:48:00.882691 (+ 34us) write_op.cc:276] Timestamp: P: 1761335275336526 usec, L: 0
1024 19:48:00.882895 (+ 204us) write_op.cc:270] Start()
1024 19:48:00.882982 (+ 87us) write_op.cc:276] Timestamp: P: 1761335275339205 usec, L: 0
1024 19:48:00.883143 (+ 161us) write_op.cc:270] Start()
1024 19:48:00.883168 (+ 25us) write_op.cc:276] Timestamp: P: 1761335275347822 usec, L: 0
1024 19:48:00.883338 (+ 170us) write_op.cc:270] Start()
1024 19:48:00.883369 (+ 31us) write_op.cc:276] Timestamp: P: 1761335275348659 usec, L: 0
1024 19:48:00.883584 (+ 215us) write_op.cc:270] Start()
1024 19:48:00.883611 (+ 27us) write_op.cc:276] Timestamp: P: 1761335275349565 usec, L: 0
1024 19:48:00.883787 (+ 176us) write_op.cc:270] Start()
1024 19:48:00.883809 (+ 22us) write_op.cc:276] Timestamp: P: 1761335275350325 usec, L: 0
1024 19:48:00.884030 (+ 221us) write_op.cc:270] Start()
1024 19:48:00.884059 (+ 29us) write_op.cc:276] Timestamp: P: 1761335275399698 usec, L: 0
1024 19:48:00.884249 (+ 190us) write_op.cc:270] Start()
1024 19:48:00.884297 (+ 48us) write_op.cc:276] Timestamp: P: 1761335275510683 usec, L: 0
1024 19:48:00.884446 (+ 149us) write_op.cc:270] Start()
1024 19:48:00.884469 (+ 23us) write_op.cc:276] Timestamp: P: 1761335275511854 usec, L: 0
1024 19:48:00.891210 (+ 6741us) write_op.cc:270] Start()
1024 19:48:00.891243 (+ 33us) write_op.cc:276] Timestamp: P: 1761335275512685 usec, L: 0
1024 19:48:00.891695 (+ 452us) write_op.cc:270] Start()
1024 19:48:00.891732 (+ 37us) write_op.cc:276] Timestamp: P: 1761335275546669 usec, L: 0
1024 19:48:00.891916 (+ 184us) write_op.cc:270] Start()
1024 19:48:00.891939 (+ 23us) write_op.cc:276] Timestamp: P: 1761335275550618 usec, L: 0
1024 19:48:00.892138 (+ 199us) write_op.cc:270] Start()
1024 19:48:00.892170 (+ 32us) write_op.cc:276] Timestamp: P: 1761335275551502 usec, L: 0
1024 19:48:00.893261 (+ 1091us) write_op.cc:270] Start()
1024 19:48:00.893293 (+ 32us) write_op.cc:276] Timestamp: P: 1761335275552396 usec, L: 0
1024 19:48:00.901995 (+ 8702us) write_op.cc:270] Start()
1024 19:48:00.902033 (+ 38us) write_op.cc:276] Timestamp: P: 1761335275553976 usec, L: 0
1024 19:48:00.902564 (+ 531us) write_op.cc:270] Start()
1024 19:48:00.902596 (+ 32us) write_op.cc:276] Timestamp: P: 1761335275556191 usec, L: 0
1024 19:48:00.902781 (+ 185us) write_op.cc:270] Start()
1024 19:48:00.902804 (+ 23us) write_op.cc:276] Timestamp: P: 1761335275556652 usec, L: 0
1024 19:48:00.908373 (+ 5569us) write_op.cc:270] Start()
1024 19:48:00.908425 (+ 52us) write_op.cc:276] Timestamp: P: 1761335275557153 usec, L: 0
1024 19:48:00.933409 (+ 24984us) write_op.cc:270] Start()
1024 19:48:00.933477 (+ 68us) write_op.cc:276] Timestamp: P: 1761335275557657 usec, L: 0
1024 19:48:00.933896 (+ 419us) write_op.cc:270] Start()
1024 19:48:00.933931 (+ 35us) write_op.cc:276] Timestamp: P: 1761335275561108 usec, L: 0
1024 19:48:00.937033 (+ 3102us) write_op.cc:270] Start()
1024 19:48:00.937060 (+ 27us) write_op.cc:276] Timestamp: P: 1761335275562728 usec, L: 0
1024 19:48:00.937282 (+ 222us) write_op.cc:270] Start()
1024 19:48:00.937310 (+ 28us) write_op.cc:276] Timestamp: P: 1761335275588730 usec, L: 0
1024 19:48:00.937492 (+ 182us) write_op.cc:270] Start()
1024 19:48:00.937519 (+ 27us) write_op.cc:276] Timestamp: P: 1761335275590396 usec, L: 0
1024 19:48:00.937719 (+ 200us) write_op.cc:270] Start()
1024 19:48:00.937745 (+ 26us) write_op.cc:276] Timestamp: P: 1761335275601101 usec, L: 0
1024 19:48:00.937906 (+ 161us) write_op.cc:270] Start()
1024 19:48:00.937930 (+ 24us) write_op.cc:276] Timestamp: P: 1761335275604484 usec, L: 0
1024 19:48:00.951890 (+ 13960us) write_op.cc:270] Start()
1024 19:48:00.951952 (+ 62us) write_op.cc:276] Timestamp: P: 1761335275610468 usec, L: 0
1024 19:48:00.952208 (+ 256us) write_op.cc:270] Start()
1024 19:48:00.952237 (+ 29us) write_op.cc:276] Timestamp: P: 1761335275613262 usec, L: 0
1024 19:48:00.952646 (+ 409us) write_op.cc:270] Start()
1024 19:48:00.953323 (+ 677us) write_op.cc:276] Timestamp: P: 1761335275644143 usec, L: 0
1024 19:48:00.953512 (+ 189us) write_op.cc:270] Start()
1024 19:48:00.953535 (+ 23us) write_op.cc:276] Timestamp: P: 1761335275657112 usec, L: 0
1024 19:48:00.953745 (+ 210us) write_op.cc:270] Start()
1024 19:48:00.953769 (+ 24us) write_op.cc:276] Timestamp: P: 1761335275662343 usec, L: 0
1024 19:48:00.953913 (+ 144us) write_op.cc:270] Start()
1024 19:48:00.953934 (+ 21us) write_op.cc:276] Timestamp: P: 1761335275668484 usec, L: 0
1024 19:48:00.954119 (+ 185us) write_op.cc:270] Start()
1024 19:48:00.954151 (+ 32us) write_op.cc:276] Timestamp: P: 1761335275672017 usec, L: 0
1024 19:48:00.954298 (+ 147us) write_op.cc:270] Start()
1024 19:48:00.954320 (+ 22us) write_op.cc:276] Timestamp: P: 1761335275673119 usec, L: 0
1024 19:48:00.954520 (+ 200us) write_op.cc:270] Start()
1024 19:48:00.954541 (+ 21us) write_op.cc:276] Timestamp: P: 1761335275768237 usec, L: 0
1024 19:48:00.954701 (+ 160us) write_op.cc:270] Start()
1024 19:48:00.954722 (+ 21us) write_op.cc:276] Timestamp: P: 1761335275771426 usec, L: 0
1024 19:48:00.954855 (+ 133us) write_op.cc:270] Start()
1024 19:48:00.954875 (+ 20us) write_op.cc:276] Timestamp: P: 1761335275775334 usec, L: 0
1024 19:48:00.955017 (+ 142us) write_op.cc:270] Start()
1024 19:48:00.955037 (+ 20us) write_op.cc:276] Timestamp: P: 1761335275817721 usec, L: 0
1024 19:48:00.955171 (+ 134us) write_op.cc:270] Start()
1024 19:48:00.955198 (+ 27us) write_op.cc:276] Timestamp: P: 1761335275832766 usec, L: 0
1024 19:48:00.955370 (+ 172us) write_op.cc:270] Start()
1024 19:48:00.955395 (+ 25us) write_op.cc:276] Timestamp: P: 1761335275838723 usec, L: 0
1024 19:48:00.955579 (+ 184us) write_op.cc:270] Start()
1024 19:48:00.955604 (+ 25us) write_op.cc:276] Timestamp: P: 1761335275843532 usec, L: 0
1024 19:48:00.955757 (+ 153us) write_op.cc:270] Start()
1024 19:48:00.955781 (+ 24us) write_op.cc:276] Timestamp: P: 1761335275844121 usec, L: 0
1024 19:48:00.955945 (+ 164us) write_op.cc:270] Start()
1024 19:48:00.955970 (+ 25us) write_op.cc:276] Timestamp: P: 1761335275847476 usec, L: 0
1024 19:48:00.956126 (+ 156us) write_op.cc:270] Start()
1024 19:48:00.956149 (+ 23us) write_op.cc:276] Timestamp: P: 1761335275848036 usec, L: 0
1024 19:48:00.956291 (+ 142us) write_op.cc:270] Start()
1024 19:48:00.956315 (+ 24us) write_op.cc:276] Timestamp: P: 1761335275848531 usec, L: 0
1024 19:48:00.956574 (+ 259us) write_op.cc:270] Start()
1024 19:48:00.956601 (+ 27us) write_op.cc:276] Timestamp: P: 1761335275849049 usec, L: 0
1024 19:48:00.956805 (+ 204us) write_op.cc:270] Start()
1024 19:48:00.956831 (+ 26us) write_op.cc:276] Timestamp: P: 1761335275858770 usec, L: 0
1024 19:48:00.956981 (+ 150us) write_op.cc:270] Start()
1024 19:48:00.957005 (+ 24us) write_op.cc:276] Timestamp: P: 1761335275859651 usec, L: 0
1024 19:48:00.957181 (+ 176us) write_op.cc:270] Start()
1024 19:48:00.957209 (+ 28us) write_op.cc:276] Timestamp: P: 1761335275860541 usec, L: 0
1024 19:48:00.957377 (+ 168us) write_op.cc:270] Start()
1024 19:48:00.957402 (+ 25us) write_op.cc:276] Timestamp: P: 1761335275861512 usec, L: 0
1024 19:48:00.957807 (+ 405us) write_op.cc:270] Start()
1024 19:48:00.957838 (+ 31us) write_op.cc:276] Timestamp: P: 1761335275902910 usec, L: 0
1024 19:48:00.958031 (+ 193us) write_op.cc:270] Start()
1024 19:48:00.958056 (+ 25us) write_op.cc:276] Timestamp: P: 1761335275903749 usec, L: 0
1024 19:48:00.958199 (+ 143us) write_op.cc:270] Start()
1024 19:48:00.958223 (+ 24us) write_op.cc:276] Timestamp: P: 1761335275919618 usec, L: 0
1024 19:48:00.958372 (+ 149us) write_op.cc:270] Start()
1024 19:48:00.958395 (+ 23us) write_op.cc:276] Timestamp: P: 1761335275927464 usec, L: 0
1024 19:48:00.958562 (+ 167us) write_op.cc:270] Start()
1024 19:48:00.958586 (+ 24us) write_op.cc:276] Timestamp: P: 1761335275928305 usec, L: 0
1024 19:48:00.958747 (+ 161us) write_op.cc:270] Start()
1024 19:48:00.958801 (+ 54us) write_op.cc:276] Timestamp: P: 1761335275929681 usec, L: 0
1024 19:48:00.958960 (+ 159us) write_op.cc:270] Start()
1024 19:48:00.959018 (+ 58us) write_op.cc:276] Timestamp: P: 1761335275930381 usec, L: 0
1024 19:48:00.959180 (+ 162us) write_op.cc:270] Start()
1024 19:48:00.959205 (+ 25us) write_op.cc:276] Timestamp: P: 1761335275982259 usec, L: 0
1024 19:48:00.959438 (+ 233us) write_op.cc:270] Start()
1024 19:48:00.959477 (+ 39us) write_op.cc:276] Timestamp: P: 1761335276034662 usec, L: 0
1024 19:48:00.959686 (+ 209us) write_op.cc:270] Start()
1024 19:48:00.959709 (+ 23us) write_op.cc:276] Timestamp: P: 1761335276035977 usec, L: 0
1024 19:48:00.960007 (+ 298us) write_op.cc:270] Start()
1024 19:48:00.960036 (+ 29us) write_op.cc:276] Timestamp: P: 1761335276066970 usec, L: 0
1024 19:48:00.960198 (+ 162us) write_op.cc:270] Start()
1024 19:48:00.960222 (+ 24us) write_op.cc:276] Timestamp: P: 1761335276067564 usec, L: 0
1024 19:48:00.960388 (+ 166us) write_op.cc:270] Start()
1024 19:48:00.960413 (+ 25us) write_op.cc:276] Timestamp: P: 1761335276071812 usec, L: 0
1024 19:48:00.960580 (+ 167us) write_op.cc:270] Start()
1024 19:48:00.960603 (+ 23us) write_op.cc:276] Timestamp: P: 1761335276078106 usec, L: 0
1024 19:48:00.982304 (+ 21701us) write_op.cc:270] Start()
1024 19:48:00.982341 (+ 37us) write_op.cc:276] Timestamp: P: 1761335276078713 usec, L: 0
1024 19:48:00.982525 (+ 184us) write_op.cc:270] Start()
1024 19:48:00.982565 (+ 40us) write_op.cc:276] Timestamp: P: 1761335276079899 usec, L: 0
1024 19:48:00.982737 (+ 172us) write_op.cc:270] Start()
1024 19:48:00.982762 (+ 25us) write_op.cc:276] Timestamp: P: 1761335276094019 usec, L: 0
1024 19:48:00.982934 (+ 172us) write_op.cc:270] Start()
1024 19:48:00.982955 (+ 21us) write_op.cc:276] Timestamp: P: 1761335276106406 usec, L: 0
1024 19:48:00.983120 (+ 165us) write_op.cc:270] Start()
1024 19:48:00.983146 (+ 26us) write_op.cc:276] Timestamp: P: 1761335276143432 usec, L: 0
1024 19:48:00.983304 (+ 158us) write_op.cc:270] Start()
1024 19:48:00.983327 (+ 23us) write_op.cc:276] Timestamp: P: 1761335276147486 usec, L: 0
1024 19:48:00.983463 (+ 136us) write_op.cc:270] Start()
1024 19:48:00.983483 (+ 20us) write_op.cc:276] Timestamp: P: 1761335276148124 usec, L: 0
1024 19:48:00.983609 (+ 126us) write_op.cc:270] Start()
1024 19:48:00.983628 (+ 19us) write_op.cc:276] Timestamp: P: 1761335276154897 usec, L: 0
1024 19:48:00.983835 (+ 207us) write_op.cc:270] Start()
1024 19:48:00.983865 (+ 30us) write_op.cc:276] Timestamp: P: 1761335276155521 usec, L: 0
1024 19:48:00.984035 (+ 170us) write_op.cc:270] Start()
1024 19:48:00.984061 (+ 26us) write_op.cc:276] Timestamp: P: 1761335276156058 usec, L: 0
1024 19:48:00.984221 (+ 160us) write_op.cc:270] Start()
1024 19:48:00.984291 (+ 70us) write_op.cc:276] Timestamp: P: 1761335276171783 usec, L: 0
1024 19:48:00.994212 (+ 9921us) write_op.cc:270] Start()
1024 19:48:00.994267 (+ 55us) write_op.cc:276] Timestamp: P: 1761335276178473 usec, L: 0
1024 19:48:00.994523 (+ 256us) write_op.cc:270] Start()
1024 19:48:00.994551 (+ 28us) write_op.cc:276] Timestamp: P: 1761335276237385 usec, L: 0
1024 19:48:00.994700 (+ 149us) write_op.cc:270] Start()
1024 19:48:00.994724 (+ 24us) write_op.cc:276] Timestamp: P: 1761335276246921 usec, L: 0
1024 19:48:00.994903 (+ 179us) write_op.cc:270] Start()
1024 19:48:00.994925 (+ 22us) write_op.cc:276] Timestamp: P: 1761335276257919 usec, L: 0
1024 19:48:00.995161 (+ 236us) write_op.cc:270] Start()
1024 19:48:00.995187 (+ 26us) write_op.cc:276] Timestamp: P: 1761335276266440 usec, L: 0
1024 19:48:00.995402 (+ 215us) write_op.cc:270] Start()
1024 19:48:00.995427 (+ 25us) write_op.cc:276] Timestamp: P: 1761335276276551 usec, L: 0
1024 19:48:00.995578 (+ 151us) write_op.cc:270] Start()
1024 19:48:00.995603 (+ 25us) write_op.cc:276] Timestamp: P: 1761335276292233 usec, L: 0
1024 19:48:00.995776 (+ 173us) write_op.cc:270] Start()
1024 19:48:00.995801 (+ 25us) write_op.cc:276] Timestamp: P: 1761335276300598 usec, L: 0
1024 19:48:00.995956 (+ 155us) write_op.cc:270] Start()
1024 19:48:00.995977 (+ 21us) write_op.cc:276] Timestamp: P: 1761335276315879 usec, L: 0
1024 19:48:00.996237 (+ 260us) write_op.cc:270] Start()
1024 19:48:00.996264 (+ 27us) write_op.cc:276] Timestamp: P: 1761335276324347 usec, L: 0
1024 19:48:00.996423 (+ 159us) write_op.cc:270] Start()
1024 19:48:00.996449 (+ 26us) write_op.cc:276] Timestamp: P: 1761335276412360 usec, L: 0
1024 19:48:00.996602 (+ 153us) write_op.cc:270] Start()
1024 19:48:00.996626 (+ 24us) write_op.cc:276] Timestamp: P: 1761335276454908 usec, L: 0
1024 19:48:00.996790 (+ 164us) write_op.cc:270] Start()
1024 19:48:00.996813 (+ 23us) write_op.cc:276] Timestamp: P: 1761335276456792 usec, L: 0
1024 19:48:00.996974 (+ 161us) write_op.cc:270] Start()
1024 19:48:00.996996 (+ 22us) write_op.cc:276] Timestamp: P: 1761335276472887 usec, L: 0
1024 19:48:00.997158 (+ 162us) write_op.cc:270] Start()
1024 19:48:00.997183 (+ 25us) write_op.cc:276] Timestamp: P: 1761335276474837 usec, L: 0
1024 19:48:00.997324 (+ 141us) write_op.cc:270] Start()
1024 19:48:00.997346 (+ 22us) write_op.cc:276] Timestamp: P: 1761335276514440 usec, L: 0
1024 19:48:00.997533 (+ 187us) write_op.cc:270] Start()
1024 19:48:00.997557 (+ 24us) write_op.cc:276] Timestamp: P: 1761335276538300 usec, L: 0
1024 19:48:00.997761 (+ 204us) write_op.cc:270] Start()
1024 19:48:00.997785 (+ 24us) write_op.cc:276] Timestamp: P: 1761335276562014 usec, L: 0
1024 19:48:00.997940 (+ 155us) write_op.cc:270] Start()
1024 19:48:00.997964 (+ 24us) write_op.cc:276] Timestamp: P: 1761335276586616 usec, L: 0
1024 19:48:00.998109 (+ 145us) write_op.cc:270] Start()
1024 19:48:00.998132 (+ 23us) write_op.cc:276] Timestamp: P: 1761335276603789 usec, L: 0
1024 19:48:00.998479 (+ 347us) write_op.cc:270] Start()
1024 19:48:00.998508 (+ 29us) write_op.cc:276] Timestamp: P: 1761335276612925 usec, L: 0
1024 19:48:01.006104 (+ 7596us) write_op.cc:270] Start()
1024 19:48:01.006167 (+ 63us) write_op.cc:276] Timestamp: P: 1761335276631552 usec, L: 0
1024 19:48:01.012358 (+ 6191us) write_op.cc:270] Start()
1024 19:48:01.012411 (+ 53us) write_op.cc:276] Timestamp: P: 1761335276647708 usec, L: 0
1024 19:48:01.012879 (+ 468us) write_op.cc:270] Start()
1024 19:48:01.012922 (+ 43us) write_op.cc:276] Timestamp: P: 1761335276648655 usec, L: 0
1024 19:48:01.013318 (+ 396us) write_op.cc:270] Start()
1024 19:48:01.013348 (+ 30us) write_op.cc:276] Timestamp: P: 1761335276649877 usec, L: 0
1024 19:48:01.013520 (+ 172us) write_op.cc:270] Start()
1024 19:48:01.013544 (+ 24us) write_op.cc:276] Timestamp: P: 1761335276650455 usec, L: 0
1024 19:48:01.013780 (+ 236us) write_op.cc:270] Start()
1024 19:48:01.013808 (+ 28us) write_op.cc:276] Timestamp: P: 1761335276660775 usec, L: 0
1024 19:48:01.013972 (+ 164us) write_op.cc:270] Start()
1024 19:48:01.013996 (+ 24us) write_op.cc:276] Timestamp: P: 1761335276678919 usec, L: 0
1024 19:48:01.014142 (+ 146us) write_op.cc:270] Start()
1024 19:48:01.014164 (+ 22us) write_op.cc:276] Timestamp: P: 1761335276742943 usec, L: 0
1024 19:48:01.014311 (+ 147us) write_op.cc:270] Start()
1024 19:48:01.014333 (+ 22us) write_op.cc:276] Timestamp: P: 1761335276743641 usec, L: 0
1024 19:48:01.014483 (+ 150us) write_op.cc:270] Start()
1024 19:48:01.014502 (+ 19us) write_op.cc:276] Timestamp: P: 1761335276744316 usec, L: 0
1024 19:48:01.014641 (+ 139us) write_op.cc:270] Start()
1024 19:48:01.014668 (+ 27us) write_op.cc:276] Timestamp: P: 1761335276744887 usec, L: 0
1024 19:48:01.014834 (+ 166us) write_op.cc:270] Start()
1024 19:48:01.014858 (+ 24us) write_op.cc:276] Timestamp: P: 1761335276745577 usec, L: 0
1024 19:48:01.015007 (+ 149us) write_op.cc:270] Start()
1024 19:48:01.015029 (+ 22us) write_op.cc:276] Timestamp: P: 1761335276746299 usec, L: 0
1024 19:48:01.015177 (+ 148us) write_op.cc:270] Start()
1024 19:48:01.015200 (+ 23us) write_op.cc:276] Timestamp: P: 1761335276747069 usec, L: 0
1024 19:48:01.015351 (+ 151us) write_op.cc:270] Start()
1024 19:48:01.015374 (+ 23us) write_op.cc:276] Timestamp: P: 1761335276777494 usec, L: 0
1024 19:48:01.015538 (+ 164us) write_op.cc:270] Start()
1024 19:48:01.015598 (+ 60us) write_op.cc:276] Timestamp: P: 1761335276778299 usec, L: 0
1024 19:48:01.015771 (+ 173us) write_op.cc:270] Start()
1024 19:48:01.015795 (+ 24us) write_op.cc:276] Timestamp: P: 1761335276792061 usec, L: 0
1024 19:48:01.015942 (+ 147us) write_op.cc:270] Start()
1024 19:48:01.015963 (+ 21us) write_op.cc:276] Timestamp: P: 1761335276797101 usec, L: 0
1024 19:48:01.016103 (+ 140us) write_op.cc:270] Start()
1024 19:48:01.016125 (+ 22us) write_op.cc:276] Timestamp: P: 1761335276802872 usec, L: 0
1024 19:48:01.016265 (+ 140us) write_op.cc:270] Start()
1024 19:48:01.016286 (+ 21us) write_op.cc:276] Timestamp: P: 1761335276813689 usec, L: 0
1024 19:48:01.016422 (+ 136us) write_op.cc:270] Start()
1024 19:48:01.016445 (+ 23us) write_op.cc:276] Timestamp: P: 1761335276821561 usec, L: 0
1024 19:48:01.016574 (+ 129us) write_op.cc:270] Start()
1024 19:48:01.016597 (+ 23us) write_op.cc:276] Timestamp: P: 1761335276826658 usec, L: 0
1024 19:48:01.016734 (+ 137us) write_op.cc:270] Start()
1024 19:48:01.016765 (+ 31us) write_op.cc:276] Timestamp: P: 1761335276830128 usec, L: 0
1024 19:48:01.016949 (+ 184us) write_op.cc:270] Start()
1024 19:48:01.016973 (+ 24us) write_op.cc:276] Timestamp: P: 1761335276839741 usec, L: 0
1024 19:48:01.017122 (+ 149us) write_op.cc:270] Start()
1024 19:48:01.017145 (+ 23us) write_op.cc:276] Timestamp: P: 1761335276875703 usec, L: 0
1024 19:48:01.017290 (+ 145us) write_op.cc:270] Start()
1024 19:48:01.017312 (+ 22us) write_op.cc:276] Timestamp: P: 1761335276887759 usec, L: 0
1024 19:48:01.017452 (+ 140us) write_op.cc:270] Start()
1024 19:48:01.017472 (+ 20us) write_op.cc:276] Timestamp: P: 1761335276888461 usec, L: 0
1024 19:48:01.020896 (+ 3424us) write_op.cc:270] Start()
1024 19:48:01.020953 (+ 57us) write_op.cc:276] Timestamp: P: 1761335276890654 usec, L: 0
1024 19:48:01.021417 (+ 464us) write_op.cc:270] Start()
1024 19:48:01.021441 (+ 24us) write_op.cc:276] Timestamp: P: 1761335276898719 usec, L: 0
1024 19:48:01.021619 (+ 178us) write_op.cc:270] Start()
1024 19:48:01.021638 (+ 19us) write_op.cc:276] Timestamp: P: 1761335276910813 usec, L: 0
1024 19:48:01.021785 (+ 147us) write_op.cc:270] Start()
1024 19:48:01.022928 (+ 1143us) write_op.cc:276] Timestamp: P: 1761335277019399 usec, L: 0
1024 19:48:01.023145 (+ 217us) write_op.cc:270] Start()
1024 19:48:01.023172 (+ 27us) write_op.cc:276] Timestamp: P: 1761335277020156 usec, L: 0
1024 19:48:01.023349 (+ 177us) write_op.cc:270] Start()
1024 19:48:01.023374 (+ 25us) write_op.cc:276] Timestamp: P: 1761335277079641 usec, L: 0
1024 19:48:01.023521 (+ 147us) write_op.cc:270] Start()
1024 19:48:01.023545 (+ 24us) write_op.cc:276] Timestamp: P: 1761335277097392 usec, L: 0
1024 19:48:01.023678 (+ 133us) write_op.cc:270] Start()
1024 19:48:01.023700 (+ 22us) write_op.cc:276] Timestamp: P: 1761335277102005 usec, L: 0
1024 19:48:01.023853 (+ 153us) write_op.cc:270] Start()
1024 19:48:01.023875 (+ 22us) write_op.cc:276] Timestamp: P: 1761335277108811 usec, L: 0
1024 19:48:01.024403 (+ 528us) write_op.cc:270] Start()
1024 19:48:01.024427 (+ 24us) write_op.cc:276] Timestamp: P: 1761335277111414 usec, L: 0
1024 19:48:01.024582 (+ 155us) write_op.cc:270] Start()
1024 19:48:01.149317 (+124735us) write_op.cc:276] Timestamp: P: 1761335277125815 usec, L: 0
1024 19:48:01.150026 (+ 709us) write_op.cc:270] Start()
1024 19:48:01.150073 (+ 47us) write_op.cc:276] Timestamp: P: 1761335277126848 usec, L: 0
1024 19:48:01.150281 (+ 208us) write_op.cc:270] Start()
1024 19:48:01.150321 (+ 40us) write_op.cc:276] Timestamp: P: 1761335277145635 usec, L: 0
1024 19:48:01.150488 (+ 167us) write_op.cc:270] Start()
1024 19:48:01.150510 (+ 22us) write_op.cc:276] Timestamp: P: 1761335277178824 usec, L: 0
1024 19:48:01.150651 (+ 141us) write_op.cc:270] Start()
1024 19:48:01.150681 (+ 30us) write_op.cc:276] Timestamp: P: 1761335277179405 usec, L: 0
1024 19:48:01.150805 (+ 124us) write_op.cc:270] Start()
1024 19:48:01.150824 (+ 19us) write_op.cc:276] Timestamp: P: 1761335277179944 usec, L: 0
1024 19:48:01.151113 (+ 289us) write_op.cc:270] Start()
1024 19:48:01.151136 (+ 23us) write_op.cc:276] Timestamp: P: 1761335277188593 usec, L: 0
1024 19:48:01.151349 (+ 213us) write_op.cc:270] Start()
1024 19:48:01.151371 (+ 22us) write_op.cc:276] Timestamp: P: 1761335277189363 usec, L: 0
1024 19:48:01.151545 (+ 174us) write_op.cc:270] Start()
1024 19:48:01.151568 (+ 23us) write_op.cc:276] Timestamp: P: 1761335277194191 usec, L: 0
1024 19:48:01.222592 (+ 71024us) write_op.cc:270] Start()
1024 19:48:01.222641 (+ 49us) write_op.cc:276] Timestamp: P: 1761335277194945 usec, L: 0
1024 19:48:01.223015 (+ 374us) write_op.cc:270] Start()
1024 19:48:01.223050 (+ 35us) write_op.cc:276] Timestamp: P: 1761335277195656 usec, L: 0
1024 19:48:01.223295 (+ 245us) write_op.cc:270] Start()
1024 19:48:01.223320 (+ 25us) write_op.cc:276] Timestamp: P: 1761335277196429 usec, L: 0
1024 19:48:01.223676 (+ 356us) write_op.cc:270] Start()
1024 19:48:01.223728 (+ 52us) write_op.cc:276] Timestamp: P: 1761335277352234 usec, L: 0
1024 19:48:01.224113 (+ 385us) write_op.cc:270] Start()
1024 19:48:01.224142 (+ 29us) write_op.cc:276] Timestamp: P: 1761335277369809 usec, L: 0
1024 19:48:01.224406 (+ 264us) write_op.cc:270] Start()
1024 19:48:01.224442 (+ 36us) write_op.cc:276] Timestamp: P: 1761335277370505 usec, L: 0
1024 19:48:01.225243 (+ 801us) write_op.cc:270] Start()
1024 19:48:01.225275 (+ 32us) write_op.cc:276] Timestamp: P: 1761335277371118 usec, L: 0
1024 19:48:01.225474 (+ 199us) write_op.cc:270] Start()
1024 19:48:01.225501 (+ 27us) write_op.cc:276] Timestamp: P: 1761335277371763 usec, L: 0
1024 19:48:01.225868 (+ 367us) write_op.cc:270] Start()
1024 19:48:01.225887 (+ 19us) write_op.cc:276] Timestamp: P: 1761335277421928 usec, L: 0
1024 19:48:01.226040 (+ 153us) write_op.cc:270] Start()
1024 19:48:01.226061 (+ 21us) write_op.cc:276] Timestamp: P: 1761335277423057 usec, L: 0
1024 19:48:01.226216 (+ 155us) write_op.cc:270] Start()
1024 19:48:01.226239 (+ 23us) write_op.cc:276] Timestamp: P: 1761335277427141 usec, L: 0
1024 19:48:01.226384 (+ 145us) write_op.cc:270] Start()
1024 19:48:01.226408 (+ 24us) write_op.cc:276] Timestamp: P: 1761335277435747 usec, L: 0
1024 19:48:01.226572 (+ 164us) write_op.cc:270] Start()
1024 19:48:01.226595 (+ 23us) write_op.cc:276] Timestamp: P: 1761335277436882 usec, L: 0
1024 19:48:01.226760 (+ 165us) write_op.cc:270] Start()
1024 19:48:01.226783 (+ 23us) write_op.cc:276] Timestamp: P: 1761335277463749 usec, L: 0
1024 19:48:01.226928 (+ 145us) write_op.cc:270] Start()
1024 19:48:01.226951 (+ 23us) write_op.cc:276] Timestamp: P: 1761335277465934 usec, L: 0
1024 19:48:01.227081 (+ 130us) write_op.cc:270] Start()
1024 19:48:01.227096 (+ 15us) write_op.cc:276] Timestamp: P: 1761335277488313 usec, L: 0
1024 19:48:01.227221 (+ 125us) write_op.cc:270] Start()
1024 19:48:01.227248 (+ 27us) write_op.cc:276] Timestamp: P: 1761335277549681 usec, L: 0
1024 19:48:01.227420 (+ 172us) write_op.cc:270] Start()
1024 19:48:01.227457 (+ 37us) write_op.cc:276] Timestamp: P: 1761335277557258 usec, L: 0
1024 19:48:01.227632 (+ 175us) write_op.cc:270] Start()
1024 19:48:01.227656 (+ 24us) write_op.cc:276] Timestamp: P: 1761335277563345 usec, L: 0
1024 19:48:01.227799 (+ 143us) write_op.cc:270] Start()
1024 19:48:01.227821 (+ 22us) write_op.cc:276] Timestamp: P: 1761335277568843 usec, L: 0
1024 19:48:01.227978 (+ 157us) write_op.cc:270] Start()
1024 19:48:01.228064 (+ 86us) write_op.cc:276] Timestamp: P: 1761335277569396 usec, L: 0
1024 19:48:01.228273 (+ 209us) write_op.cc:270] Start()
1024 19:48:01.228298 (+ 25us) write_op.cc:276] Timestamp: P: 1761335277569908 usec, L: 0
1024 19:48:01.228461 (+ 163us) write_op.cc:270] Start()
1024 19:48:01.228485 (+ 24us) write_op.cc:276] Timestamp: P: 1761335277577543 usec, L: 0
1024 19:48:01.228645 (+ 160us) write_op.cc:270] Start()
1024 19:48:01.228668 (+ 23us) write_op.cc:276] Timestamp: P: 1761335277600287 usec, L: 0
1024 19:48:01.228812 (+ 144us) write_op.cc:270] Start()
1024 19:48:01.228836 (+ 24us) write_op.cc:276] Timestamp: P: 1761335277605339 usec, L: 0
1024 19:48:01.228973 (+ 137us) write_op.cc:270] Start()
1024 19:48:01.228997 (+ 24us) write_op.cc:276] Timestamp: P: 1761335277606342 usec, L: 0
1024 19:48:01.229148 (+ 151us) write_op.cc:270] Start()
1024 19:48:01.229168 (+ 20us) write_op.cc:276] Timestamp: P: 1761335277640397 usec, L: 0
1024 19:48:01.229312 (+ 144us) write_op.cc:270] Start()
1024 19:48:01.229350 (+ 38us) write_op.cc:276] Timestamp: P: 1761335277641274 usec, L: 0
1024 19:48:01.229508 (+ 158us) write_op.cc:270] Start()
1024 19:48:01.229532 (+ 24us) write_op.cc:276] Timestamp: P: 1761335277644562 usec, L: 0
1024 19:48:01.504061 (+274529us) write_op.cc:270] Start()
1024 19:48:01.504097 (+ 36us) write_op.cc:276] Timestamp: P: 1761335277654602 usec, L: 0
1024 19:48:01.509145 (+ 5048us) write_op.cc:270] Start()
1024 19:48:01.509187 (+ 42us) write_op.cc:276] Timestamp: P: 1761335277655365 usec, L: 0
1024 19:48:01.509422 (+ 235us) write_op.cc:270] Start()
1024 19:48:01.509452 (+ 30us) write_op.cc:276] Timestamp: P: 1761335277657140 usec, L: 0
1024 19:48:01.509664 (+ 212us) write_op.cc:270] Start()
1024 19:48:01.509692 (+ 28us) write_op.cc:276] Timestamp: P: 1761335277662706 usec, L: 0
1024 19:48:01.509879 (+ 187us) write_op.cc:270] Start()
1024 19:48:01.509903 (+ 24us) write_op.cc:276] Timestamp: P: 1761335277663299 usec, L: 0
1024 19:48:01.510041 (+ 138us) write_op.cc:270] Start()
1024 19:48:01.510065 (+ 24us) write_op.cc:276] Timestamp: P: 1761335277663841 usec, L: 0
1024 19:48:01.510215 (+ 150us) write_op.cc:270] Start()
1024 19:48:01.510236 (+ 21us) write_op.cc:276] Timestamp: P: 1761335277735868 usec, L: 0
1024 19:48:01.510379 (+ 143us) write_op.cc:270] Start()
1024 19:48:01.510412 (+ 33us) write_op.cc:276] Timestamp: P: 1761335277736534 usec, L: 0
1024 19:48:01.510917 (+ 505us) write_op.cc:270] Start()
1024 19:48:01.510946 (+ 29us) write_op.cc:276] Timestamp: P: 1761335277738710 usec, L: 0
1024 19:48:01.511116 (+ 170us) write_op.cc:270] Start()
1024 19:48:01.511142 (+ 26us) write_op.cc:276] Timestamp: P: 1761335277746345 usec, L: 0
1024 19:48:01.511276 (+ 134us) write_op.cc:270] Start()
1024 19:48:01.511297 (+ 21us) write_op.cc:276] Timestamp: P: 1761335277749922 usec, L: 0
1024 19:48:01.511456 (+ 159us) write_op.cc:270] Start()
1024 19:48:01.511478 (+ 22us) write_op.cc:276] Timestamp: P: 1761335277859759 usec, L: 0
1024 19:48:01.511620 (+ 142us) write_op.cc:270] Start()
1024 19:48:01.511643 (+ 23us) write_op.cc:276] Timestamp: P: 1761335277860462 usec, L: 0
1024 19:48:01.511801 (+ 158us) write_op.cc:270] Start()
1024 19:48:01.511824 (+ 23us) write_op.cc:276] Timestamp: P: 1761335277866154 usec, L: 0
1024 19:48:01.511953 (+ 129us) write_op.cc:270] Start()
1024 19:48:01.511972 (+ 19us) write_op.cc:276] Timestamp: P: 1761335277884850 usec, L: 0
1024 19:48:01.512119 (+ 147us) write_op.cc:270] Start()
1024 19:48:01.512139 (+ 20us) write_op.cc:276] Timestamp: P: 1761335277891063 usec, L: 0
1024 19:48:01.512277 (+ 138us) write_op.cc:270] Start()
1024 19:48:01.512297 (+ 20us) write_op.cc:276] Timestamp: P: 1761335277900411 usec, L: 0
1024 19:48:01.513003 (+ 706us) write_op.cc:270] Start()
1024 19:48:01.513027 (+ 24us) write_op.cc:276] Timestamp: P: 1761335277906353 usec, L: 0
1024 19:48:01.513179 (+ 152us) write_op.cc:270] Start()
1024 19:48:01.513231 (+ 52us) write_op.cc:276] Timestamp: P: 1761335277907006 usec, L: 0
1024 19:48:01.513507 (+ 276us) write_op.cc:270] Start()
1024 19:48:01.513532 (+ 25us) write_op.cc:276] Timestamp: P: 1761335277908524 usec, L: 0
1024 19:48:01.513726 (+ 194us) write_op.cc:270] Start()
1024 19:48:01.513759 (+ 33us) write_op.cc:276] Timestamp: P: 1761335277916817 usec, L: 0
1024 19:48:01.514344 (+ 585us) write_op.cc:270] Start()
1024 19:48:01.514369 (+ 25us) write_op.cc:276] Timestamp: P: 1761335
W20251024 19:48:03.795761 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edbcecddf69d4de8af2cf58ac7b12902 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.796475 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b6fb408e15354fb7b55bf182bb981be3" } new_table_name: "default.Table_dbe9a03f95bf479490b2015a0c6afe20" modify_external_catalogs: true
I20251024 19:48:03.798447 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_631ebe0a47914d38854b5d13f0ea4af4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.799175 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_631ebe0a47914d38854b5d13f0ea4af4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.800721 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_11634cbabbd6478484a0aed448ae9521" } modify_external_catalogs: true
I20251024 19:48:03.800935 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_11634cbabbd6478484a0aed448ae9521" } modify_external_catalogs: true
I20251024 19:48:03.801753 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ea0c3c7222394ffda2136fe114e6760b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.802433 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea0c3c7222394ffda2136fe114e6760b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.804225 1768 tablet_bootstrap.cc:492] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:03.804422 1768 ts_tablet_manager.cc:1403] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.008s
W20251024 19:48:03.805192 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 46578.2us
I20251024 19:48:03.805615 2020 raft_consensus.cc:2804] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:03.805644 1768 raft_consensus.cc:359] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.805851 1768 raft_consensus.cc:385] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.805994 1768 raft_consensus.cc:740] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.806272 1768 consensus_queue.cc:260] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.807022 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0813796576a2420e8adafd30070bc747" } new_table_name: "default.Table_df7191b6e1d94c5fae650a13660bf68b" modify_external_catalogs: true
W20251024 19:48:03.807523 1447 leader_election.cc:343] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: Tablet not RUNNING: STOPPED
I20251024 19:48:03.807796 1745 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:48:03.808073 1745 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: This master's current role is: LEADER
I20251024 19:48:03.808151 2020 raft_consensus.cc:493] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:03.808326 2020 raft_consensus.cc:3060] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:03.810832 1768 ts_tablet_manager.cc:1434] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20251024 19:48:03.813560 1371 raft_consensus.cc:2468] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
W20251024 19:48:03.819053 1308 leader_election.cc:343] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.819212 1308 leader_election.cc:304] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:03.819695 2154 raft_consensus.cc:2749] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:03.820263 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d1933bb728845abad5e6b09b24868f0" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:03.820508 1511 raft_consensus.cc:2468] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:03.820926 1768 tablet_bootstrap.cc:492] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:03.821974 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet a48249f1824a40b087b80a563816d485 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.822088 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f346a90a9efe442f82750336e3893deb" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:03.822237 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ba7eebe9102b4e27ad8227e35ed754ba" } new_table_name: "default.Table_3114365d011943a389d8d35215265bbf" modify_external_catalogs: true
W20251024 19:48:03.823069 1447 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 67509.5us
I20251024 19:48:03.823251 1768 tablet_bootstrap.cc:654] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.823328 2154 consensus_queue.cc:1048] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:03.824875 1308 leader_election.cc:304] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
W20251024 19:48:03.825254 1308 leader_election.cc:343] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.825382 1308 leader_election.cc:304] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:03.825899 2158 raft_consensus.cc:2804] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:03.825999 2158 raft_consensus.cc:493] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:03.826033 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 346) took 1812 ms. Trace:
I20251024 19:48:03.826108 2158 raft_consensus.cc:3060] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:03.826166 1776 rpcz_store.cc:276] 1024 19:48:02.013548 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.017274 (+ 3726us) service_pool.cc:225] Handling call
1024 19:48:03.578078 (+1560804us) ts_tablet_manager.cc:1151] Deleting tablet c292ce055def4964ab143252c6b2b87e
1024 19:48:03.578090 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.661811 (+ 83721us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.751683 (+ 89872us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.825992 (+ 74309us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1551512}
I20251024 19:48:03.829146 2130 raft_consensus.cc:2749] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:03.829331 2130 consensus_queue.cc:1048] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:03.829954 1766 raft_consensus.cc:359] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.830132 1766 raft_consensus.cc:385] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.831709 1766 raft_consensus.cc:740] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.832010 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 2602335327fc489fa154056c5ef5138f with delete_type TABLET_DATA_DELETED (replaced by tablet 74029a7d7cdc4503a833c1758887b684) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.831965 1766 consensus_queue.cc:260] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.832715 1766 ts_tablet_manager.cc:1434] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.049s user 0.001s sys 0.001s
I20251024 19:48:03.832990 1766 tablet_bootstrap.cc:492] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:03.834882 1768 tablet_bootstrap.cc:492] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:03.834882 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet ee68243b4a664575b679950603bb250e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.835036 1768 ts_tablet_manager.cc:1403] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
W20251024 19:48:03.835250 1444 consensus_peers.cc:597] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 369cebfda6ee43638b79f8652fa55855. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:03.835718 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 320) took 2162 ms. Trace:
I20251024 19:48:03.835800 1768 raft_consensus.cc:359] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.835991 1768 raft_consensus.cc:385] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.836071 1768 raft_consensus.cc:740] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.836359 1768 consensus_queue.cc:260] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.837179 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 347) took 1777 ms. Trace:
I20251024 19:48:03.837208 1768 ts_tablet_manager.cc:1434] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:03.837634 1768 tablet_bootstrap.cc:492] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:03.837483 1776 rpcz_store.cc:276] 1024 19:48:02.059370 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.062476 (+ 3106us) service_pool.cc:225] Handling call
1024 19:48:03.828570 (+1766094us) ts_tablet_manager.cc:1151] Deleting tablet 3e6db58fac5e4af9b27bde6c223aa845
1024 19:48:03.828580 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.837127 (+ 8547us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1764919}
I20251024 19:48:03.837893 1447 leader_election.cc:304] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
W20251024 19:48:03.838110 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3e6db58fac5e4af9b27bde6c223aa845 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e6db58fac5e4af9b27bde6c223aa845 already in progress: creating tablet
I20251024 19:48:03.838163 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 348) took 1734 ms. Trace:
I20251024 19:48:03.838280 1776 rpcz_store.cc:276] 1024 19:48:02.103277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.103508 (+ 231us) service_pool.cc:225] Handling call
1024 19:48:03.837993 (+1734485us) ts_tablet_manager.cc:1151] Deleting tablet 2496f103436047dea5dabb577651249f
1024 19:48:03.838003 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.838116 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1734215}
I20251024 19:48:03.838771 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 349) took 1711 ms. Trace:
W20251024 19:48:03.838855 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2496f103436047dea5dabb577651249f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2496f103436047dea5dabb577651249f already in progress: creating tablet
I20251024 19:48:03.838892 1776 rpcz_store.cc:276] 1024 19:48:02.127319 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.127388 (+ 69us) service_pool.cc:225] Handling call
1024 19:48:03.838630 (+1711242us) ts_tablet_manager.cc:1151] Deleting tablet 96a6e0a4882c42399fd07f06ac21aa3f
1024 19:48:03.838638 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.838732 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1710770}
I20251024 19:48:03.839264 1776 tablet_replica.cc:333] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:48:03.839278 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 96a6e0a4882c42399fd07f06ac21aa3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96a6e0a4882c42399fd07f06ac21aa3f already in progress: creating tablet
I20251024 19:48:03.839454 1776 raft_consensus.cc:2243] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:03.839625 1776 raft_consensus.cc:2272] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:03.840484 1776 ts_tablet_manager.cc:1916] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:03.840404 1775 rpcz_store.cc:276] 1024 19:48:01.669489 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.669860 (+ 371us) service_pool.cc:225] Handling call
1024 19:48:03.551812 (+1881952us) ts_tablet_manager.cc:1151] Deleting tablet 9820e68ad5264df1a79b22f3b5369c5a
1024 19:48:03.551824 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.697805 (+145981us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.782323 (+ 84518us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.831490 (+ 49167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4,"tablet-delete.queue_time_us":1881689}
I20251024 19:48:03.840837 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 321) took 2150 ms. Trace:
I20251024 19:48:03.842057 1775 rpcz_store.cc:276] 1024 19:48:01.690090 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.690146 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:03.840705 (+2150559us) ts_tablet_manager.cc:1151] Deleting tablet 62667092d7e244cca1020ec220ea9124
1024 19:48:03.840714 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.840801 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2150284}
I20251024 19:48:03.842358 1775 tablet_replica.cc:333] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:03.842967 1775 raft_consensus.cc:2243] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:03.843137 1775 raft_consensus.cc:2272] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:03.844538 1775 ts_tablet_manager.cc:1916] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:03.846027 1767 tablet_bootstrap.cc:492] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:03.846127 1768 tablet_bootstrap.cc:654] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.846164 1767 ts_tablet_manager.cc:1403] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.131s user 0.000s sys 0.006s
I20251024 19:48:03.846374 2141 raft_consensus.cc:2804] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:03.846812 2141 raft_consensus.cc:697] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:03.847245 2141 consensus_queue.cc:237] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.849889 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 9820e68ad5264df1a79b22f3b5369c5a (table default.Table_0680f9139a8a436a80fce09c095912ce [id=071147c9baed46f8a30f4c2fbc0fbfaa]) successfully deleted
W20251024 19:48:03.850244 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 62667092d7e244cca1020ec220ea9124 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62667092d7e244cca1020ec220ea9124 already in progress: creating tablet
I20251024 19:48:03.851704 1767 raft_consensus.cc:359] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:03.852473 1767 raft_consensus.cc:385] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.852830 1767 raft_consensus.cc:740] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.853636 1767 consensus_queue.cc:260] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:03.855533 1767 ts_tablet_manager.cc:1434] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251024 19:48:03.856457 1767 tablet_bootstrap.cc:492] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:03.856846 1766 tablet_bootstrap.cc:654] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.858801 1767 tablet_bootstrap.cc:654] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.862975 2173 raft_consensus.cc:493] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.863116 2173 raft_consensus.cc:515] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.863843 2173 leader_election.cc:290] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:03.864780 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48249f1824a40b087b80a563816d485" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:03.865507 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48249f1824a40b087b80a563816d485" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:03.865563 1444 leader_election.cc:343] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.865754 1371 raft_consensus.cc:2468] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:03.866197 1447 leader_election.cc:304] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:03.866606 2173 raft_consensus.cc:2804] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:03.866729 2173 raft_consensus.cc:493] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:03.866834 2173 raft_consensus.cc:3060] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:03.884060 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46c76d86a68c40e18cc176b4776db266 has no consensus state
W20251024 19:48:03.886526 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15de0d8aa9e545c09233186bbe361b97 has no consensus state
W20251024 19:48:03.886899 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 73278db2097d4001a538d62dacb97c9b consensus state has no leader
I20251024 19:48:03.889057 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 15de0d8aa9e545c09233186bbe361b97 (DEFAULT_TABLE table=default.Table_df7191b6e1d94c5fae650a13660bf68b [id=8fe376f8878c428abf0aafa824c5026b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.890198 1341 tablet_service.cc:1505] Processing CreateTablet for tablet d0e5f7c28f0c47a88ca9872db2184ad0 (DEFAULT_TABLE table=default.Table_34772bf3583943e4a3ffa064e9dd1333 [id=b77f1765733b4a669da663c9be449938]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.891414 1184 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet c04082bd1b444b468c177bef239701c6 (table default.Table_24200cd6f19d460cae963afb9b7f8724 [id=a2789faf4abe4c09b0a7515d941831a6]). Expected version 2 got 0
I20251024 19:48:03.892853 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet f346a90a9efe442f82750336e3893deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:03.893182 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet d38154d3ffd34f4b8d7305b1c23a4fae from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:03.893498 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 15de0d8aa9e545c09233186bbe361b97 (DEFAULT_TABLE table=default.Table_df7191b6e1d94c5fae650a13660bf68b [id=8fe376f8878c428abf0aafa824c5026b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.894028 1487 tablet_service.cc:1505] Processing CreateTablet for tablet d0e5f7c28f0c47a88ca9872db2184ad0 (DEFAULT_TABLE table=default.Table_34772bf3583943e4a3ffa064e9dd1333 [id=b77f1765733b4a669da663c9be449938]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.894587 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 15de0d8aa9e545c09233186bbe361b97 (DEFAULT_TABLE table=default.Table_df7191b6e1d94c5fae650a13660bf68b [id=8fe376f8878c428abf0aafa824c5026b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.895256 1625 tablet_service.cc:1505] Processing CreateTablet for tablet d0e5f7c28f0c47a88ca9872db2184ad0 (DEFAULT_TABLE table=default.Table_34772bf3583943e4a3ffa064e9dd1333 [id=b77f1765733b4a669da663c9be449938]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.897333 2146 raft_consensus.cc:493] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.897511 2146 raft_consensus.cc:515] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.898303 2146 leader_election.cc:290] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:03.898979 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c04082bd1b444b468c177bef239701c6" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:03.899201 1371 raft_consensus.cc:2468] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:03.900043 1586 leader_election.cc:304] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:03.900524 2146 raft_consensus.cc:2804] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:03.900763 2146 raft_consensus.cc:493] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:03.901001 2146 raft_consensus.cc:3060] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:03.901372 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a2f9ff12c83949f89a0a0442e8f8189e" } modify_external_catalogs: true
I20251024 19:48:03.901676 1768 tablet_bootstrap.cc:492] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:03.901625 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a2f9ff12c83949f89a0a0442e8f8189e" } modify_external_catalogs: true
I20251024 19:48:03.901813 1768 ts_tablet_manager.cc:1403] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.064s user 0.010s sys 0.017s
I20251024 19:48:03.902572 1768 raft_consensus.cc:359] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.903661 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet ad81d80697984ff1834a7d795e0f4b9b from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:03.902760 1768 raft_consensus.cc:385] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:03.906230 1768 raft_consensus.cc:740] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:03.906491 1768 consensus_queue.cc:260] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.907172 1768 ts_tablet_manager.cc:1434] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251024 19:48:03.912027 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 774a2de2e3764c54b72acfb952ed3e11 (DEFAULT_TABLE table=default.Table_7e15b1eebd3443548f58b749626a8679 [id=1a8ca3395d7b471fa354154f2468080f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.912395 2130 raft_consensus.cc:493] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.912578 2130 raft_consensus.cc:515] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.912815 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 46c76d86a68c40e18cc176b4776db266 (DEFAULT_TABLE table=default.Table_dbe9a03f95bf479490b2015a0c6afe20 [id=51716cd9fedf415bb03a962bb4235b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.913218 2130 leader_election.cc:290] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:03.913791 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 774a2de2e3764c54b72acfb952ed3e11 (DEFAULT_TABLE table=default.Table_7e15b1eebd3443548f58b749626a8679 [id=1a8ca3395d7b471fa354154f2468080f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.913864 1768 tablet_bootstrap.cc:492] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:03.914652 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5473699c1b647f8b86ee43fdb56bdfd" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:03.915635 1305 leader_election.cc:343] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:03.915884 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 44ee03ede02c4bdab706fde910978332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:03.916934 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 7172a93a5bec4fa1b63282ba63beec0c (DEFAULT_TABLE table=default.Table_c8ab8e29b5a94252a5395f467f5e130f [id=c852fde504af48d6adb89c89c9a8b213]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.917943 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 373aa543e1284fbc945ca802d9eddf59 (DEFAULT_TABLE table=default.Table_aa4ebf0155944ee2bfba2c23eacb5655 [id=133851875faa426391a9a8e2a875d771]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.918044 1774 ts_tablet_manager.cc:1929] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:03.918188 1774 log.cc:1199] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/346383e85cc64b3a8da70a71868ceccc
I20251024 19:48:03.918610 1774 ts_tablet_manager.cc:1950] T 346383e85cc64b3a8da70a71868ceccc P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:03.918931 2146 raft_consensus.cc:515] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.919229 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 7172a93a5bec4fa1b63282ba63beec0c (DEFAULT_TABLE table=default.Table_c8ab8e29b5a94252a5395f467f5e130f [id=c852fde504af48d6adb89c89c9a8b213]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.919610 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7172a93a5bec4fa1b63282ba63beec0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.919586 2146 leader_election.cc:290] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:03.919884 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0e5f7c28f0c47a88ca9872db2184ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.920079 1768 tablet_bootstrap.cc:654] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:03.920207 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 46c76d86a68c40e18cc176b4776db266 (DEFAULT_TABLE table=default.Table_dbe9a03f95bf479490b2015a0c6afe20 [id=51716cd9fedf415bb03a962bb4235b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.920488 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15de0d8aa9e545c09233186bbe361b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.920974 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 774a2de2e3764c54b72acfb952ed3e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.921257 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 46c76d86a68c40e18cc176b4776db266 (DEFAULT_TABLE table=default.Table_dbe9a03f95bf479490b2015a0c6afe20 [id=51716cd9fedf415bb03a962bb4235b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.922106 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 7172a93a5bec4fa1b63282ba63beec0c (DEFAULT_TABLE table=default.Table_c8ab8e29b5a94252a5395f467f5e130f [id=c852fde504af48d6adb89c89c9a8b213]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.922737 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 774a2de2e3764c54b72acfb952ed3e11 (DEFAULT_TABLE table=default.Table_7e15b1eebd3443548f58b749626a8679 [id=1a8ca3395d7b471fa354154f2468080f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.926942 1184 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 2346c03802a245a2af8c086b626d660d (table default.Table_03760ade20c7464f81dd428a3eb734fb [id=aadca5a1b6d4494aa37f8a8cc76ad12a]). Expected version 5 got 0
I20251024 19:48:03.928892 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c04082bd1b444b468c177bef239701c6" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:03.929797 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c76d86a68c40e18cc176b4776db266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.930498 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 373aa543e1284fbc945ca802d9eddf59 (DEFAULT_TABLE table=default.Table_aa4ebf0155944ee2bfba2c23eacb5655 [id=133851875faa426391a9a8e2a875d771]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.930619 1510 raft_consensus.cc:2468] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:03.930996 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 373aa543e1284fbc945ca802d9eddf59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.934131 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 4c54e569156042fca5e9ab0feaedf632 (DEFAULT_TABLE table=default.Table_5c266c8c172d4d08847b98402d5d0ffb [id=e9c111066e3b4674bbf32a05f062fb36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.937379 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 7deb97be82964ea0b2fe9caa867a6460 (DEFAULT_TABLE table=default.Table_ecf8ed863a5b41fb888927d73b57c8a6 [id=041b01b6f1cd43839efe5ac538461b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.938238 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 7da1601a87524ef1a8b5069ba6823843 (DEFAULT_TABLE table=default.Table_7dc53b918ca7443b807133df217c0f75 [id=9ce6ac48924e45e3a482f71740edb4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.939260 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 9684072368f14cc2942b4e240f90b0f2 (DEFAULT_TABLE table=default.Table_a1c0c56a2bd5452293c935a6aecb39e3 [id=a50b7146b9ca4b869985a6f5cbad6036]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.940459 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2c02fd9cab9b479d8876a6dcbd586b17" } modify_external_catalogs: true
I20251024 19:48:03.940723 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2c02fd9cab9b479d8876a6dcbd586b17" } modify_external_catalogs: true
I20251024 19:48:03.941817 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 4c54e569156042fca5e9ab0feaedf632 (DEFAULT_TABLE table=default.Table_5c266c8c172d4d08847b98402d5d0ffb [id=e9c111066e3b4674bbf32a05f062fb36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.942366 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c54e569156042fca5e9ab0feaedf632. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.944480 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c04082bd1b444b468c177bef239701c6" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:03.944742 1510 raft_consensus.cc:3060] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:03.945060 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c04082bd1b444b468c177bef239701c6" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:03.945339 1371 raft_consensus.cc:3060] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:03.945569 1623 tablet_service.cc:1505] Processing CreateTablet for tablet d731403fbe0b4f92ac5899e13ea7bb19 (DEFAULT_TABLE table=default.Table_0dfdcd7f531e4ddca00916bf1dcec977 [id=34c45c3b9f52491aaf5846b7b674d449]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.946178 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 7da1601a87524ef1a8b5069ba6823843 (DEFAULT_TABLE table=default.Table_7dc53b918ca7443b807133df217c0f75 [id=9ce6ac48924e45e3a482f71740edb4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.946798 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da1601a87524ef1a8b5069ba6823843. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.946934 2158 raft_consensus.cc:515] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:03.949133 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 373aa543e1284fbc945ca802d9eddf59 (DEFAULT_TABLE table=default.Table_aa4ebf0155944ee2bfba2c23eacb5655 [id=133851875faa426391a9a8e2a875d771]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.949627 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 7deb97be82964ea0b2fe9caa867a6460 (DEFAULT_TABLE table=default.Table_ecf8ed863a5b41fb888927d73b57c8a6 [id=041b01b6f1cd43839efe5ac538461b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.949816 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 4c54e569156042fca5e9ab0feaedf632 (DEFAULT_TABLE table=default.Table_5c266c8c172d4d08847b98402d5d0ffb [id=e9c111066e3b4674bbf32a05f062fb36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.950181 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 7da1601a87524ef1a8b5069ba6823843 (DEFAULT_TABLE table=default.Table_7dc53b918ca7443b807133df217c0f75 [id=9ce6ac48924e45e3a482f71740edb4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.950134 2020 raft_consensus.cc:515] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:03.950851 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 9684072368f14cc2942b4e240f90b0f2 (DEFAULT_TABLE table=default.Table_a1c0c56a2bd5452293c935a6aecb39e3 [id=a50b7146b9ca4b869985a6f5cbad6036]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.951046 2020 leader_election.cc:290] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:03.951583 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 774a2de2e3764c54b72acfb952ed3e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.951853 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9684072368f14cc2942b4e240f90b0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.952088 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c76d86a68c40e18cc176b4776db266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.952289 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 373aa543e1284fbc945ca802d9eddf59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.952512 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0e5f7c28f0c47a88ca9872db2184ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.952827 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d731403fbe0b4f92ac5899e13ea7bb19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.953162 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15de0d8aa9e545c09233186bbe361b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.953343 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c54e569156042fca5e9ab0feaedf632. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.953773 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0e5f7c28f0c47a88ca9872db2184ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.953951 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7172a93a5bec4fa1b63282ba63beec0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.954294 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c54e569156042fca5e9ab0feaedf632. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.954497 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7172a93a5bec4fa1b63282ba63beec0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.954718 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15de0d8aa9e545c09233186bbe361b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.954931 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9684072368f14cc2942b4e240f90b0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.955186 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 373aa543e1284fbc945ca802d9eddf59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.955425 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c76d86a68c40e18cc176b4776db266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.955915 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7deb97be82964ea0b2fe9caa867a6460. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.956382 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da1601a87524ef1a8b5069ba6823843. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.956964 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7deb97be82964ea0b2fe9caa867a6460. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.957515 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da1601a87524ef1a8b5069ba6823843. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.962869 1611 tablet_service.cc:1505] Processing CreateTablet for tablet af43ffe097e14c4faffed95e207d0893 (DEFAULT_TABLE table=default.Table_61a8b015b11f4effa0245b9737c3c2ff [id=b2b7cd3fb1cc45009d1656ad27a443c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.963366 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af43ffe097e14c4faffed95e207d0893. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.964238 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 9684072368f14cc2942b4e240f90b0f2 (DEFAULT_TABLE table=default.Table_a1c0c56a2bd5452293c935a6aecb39e3 [id=a50b7146b9ca4b869985a6f5cbad6036]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.964783 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9684072368f14cc2942b4e240f90b0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.969452 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 7deb97be82964ea0b2fe9caa867a6460 (DEFAULT_TABLE table=default.Table_ecf8ed863a5b41fb888927d73b57c8a6 [id=041b01b6f1cd43839efe5ac538461b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.971486 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 260) took 3299 ms. Trace:
I20251024 19:48:03.971666 1774 rpcz_store.cc:276] 1024 19:48:00.650704 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.650750 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:03.752263 (+3101513us) ts_tablet_manager.cc:1151] Deleting tablet 346383e85cc64b3a8da70a71868ceccc
1024 19:48:03.752275 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.813987 (+ 61712us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.917910 (+103923us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:03.949791 (+ 31881us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3101312}
I20251024 19:48:03.974149 1774 tablet_replica.cc:333] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:03.974392 1774 raft_consensus.cc:2243] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:03.975435 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d1933bb728845abad5e6b09b24868f0" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:03.976015 1774 raft_consensus.cc:2272] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:03.978024 1447 leader_election.cc:343] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: e20881e400cf4fac914b70afc79c084f
I20251024 19:48:03.980063 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 346383e85cc64b3a8da70a71868ceccc (table default.Table_acb41c25b40742ef92f4bb9d45ada0a3 [id=44c9c172ae1242d3a7ac272857209ff8]) successfully deleted
I20251024 19:48:03.981300 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ef3fdce598e74165bddac418dc92c20e" } modify_external_catalogs: true
I20251024 19:48:03.981523 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ef3fdce598e74165bddac418dc92c20e" } modify_external_catalogs: true
I20251024 19:48:03.983006 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_29c99dcba4ab4430aa0b7cb3c70621e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:03.983111 1485 tablet_service.cc:1505] Processing CreateTablet for tablet d731403fbe0b4f92ac5899e13ea7bb19 (DEFAULT_TABLE table=default.Table_0dfdcd7f531e4ddca00916bf1dcec977 [id=34c45c3b9f52491aaf5846b7b674d449]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.983697 1486 tablet_service.cc:1505] Processing CreateTablet for tablet af43ffe097e14c4faffed95e207d0893 (DEFAULT_TABLE table=default.Table_61a8b015b11f4effa0245b9737c3c2ff [id=b2b7cd3fb1cc45009d1656ad27a443c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:03.988188 2130 raft_consensus.cc:493] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:03.988551 2158 leader_election.cc:290] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:03.989070 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29c99dcba4ab4430aa0b7cb3c70621e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.989629 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 774a2de2e3764c54b72acfb952ed3e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:03.990859 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ae2a66ac44f249a2b5335fa64ff1c99c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.992219 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae2a66ac44f249a2b5335fa64ff1c99c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.993633 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7abb98af07f64d6899a9c6d7b2d84aa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:03.994192 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7abb98af07f64d6899a9c6d7b2d84aa2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:03.999290 1774 ts_tablet_manager.cc:1916] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:04.004635 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7deb97be82964ea0b2fe9caa867a6460. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:04.007323 1305 leader_election.cc:343] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.008338 1348 tablet_service.cc:1505] Processing CreateTablet for tablet d731403fbe0b4f92ac5899e13ea7bb19 (DEFAULT_TABLE table=default.Table_0dfdcd7f531e4ddca00916bf1dcec977 [id=34c45c3b9f52491aaf5846b7b674d449]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.008785 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d731403fbe0b4f92ac5899e13ea7bb19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.009510 1347 tablet_service.cc:1505] Processing CreateTablet for tablet af43ffe097e14c4faffed95e207d0893 (DEFAULT_TABLE table=default.Table_61a8b015b11f4effa0245b9737c3c2ff [id=b2b7cd3fb1cc45009d1656ad27a443c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.009933 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af43ffe097e14c4faffed95e207d0893. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.010537 2130 raft_consensus.cc:515] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.012058 1648 raft_consensus.cc:3060] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.019261 1775 ts_tablet_manager.cc:1929] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:04.020273 1508 raft_consensus.cc:2468] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:04.027987 1586 leader_election.cc:304] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:04.028487 2146 raft_consensus.cc:2804] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:04.028637 2146 raft_consensus.cc:697] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:04.028928 2146 consensus_queue.cc:237] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.025696 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d731403fbe0b4f92ac5899e13ea7bb19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.033731 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af43ffe097e14c4faffed95e207d0893. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.019559 1775 log.cc:1199] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/f91316c030dc4f7da27977a9dde01f4b
I20251024 19:48:04.039856 1775 ts_tablet_manager.cc:1950] T f91316c030dc4f7da27977a9dde01f4b P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:04.041005 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3b4e85f741849b2a6a3e8574396ff88" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:04.053689 2130 leader_election.cc:290] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:04.054589 1305 leader_election.cc:343] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.059662 2229 tablet.cc:1722] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251024 19:48:04.113238 2173 raft_consensus.cc:515] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.113962 1371 raft_consensus.cc:2468] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:04.114053 2173 leader_election.cc:290] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:04.114567 1586 leader_election.cc:304] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:04.114917 2146 raft_consensus.cc:2804] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:04.115059 2146 raft_consensus.cc:697] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:04.115341 2146 consensus_queue.cc:237] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.115847 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 322) took 2414 ms. Trace:
I20251024 19:48:04.115935 1775 rpcz_store.cc:276] 1024 19:48:01.701372 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.701479 (+ 107us) service_pool.cc:225] Handling call
1024 19:48:03.842302 (+2140823us) ts_tablet_manager.cc:1151] Deleting tablet f91316c030dc4f7da27977a9dde01f4b
1024 19:48:03.842310 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.953397 (+111087us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.019144 (+ 65747us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.115792 (+ 96648us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2139697}
I20251024 19:48:04.116289 1775 tablet_replica.cc:333] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:04.116458 1775 raft_consensus.cc:2243] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:04.116786 1775 raft_consensus.cc:2272] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:04.118743 1775 ts_tablet_manager.cc:1916] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:04.119072 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48249f1824a40b087b80a563816d485" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:04.119290 1371 raft_consensus.cc:3060] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.159617 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet f91316c030dc4f7da27977a9dde01f4b (table default.Table_cdf2e8872b724c7aa86d5091eb52fb1c [id=a402387542654ccf8e0b04ca8ecebb1a]) successfully deleted
I20251024 19:48:04.194671 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e20881e400cf4fac914b70afc79c084f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:04.194985 1650 raft_consensus.cc:3060] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.201510 1766 tablet_bootstrap.cc:492] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:04.201736 1766 ts_tablet_manager.cc:1403] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.369s user 0.002s sys 0.009s
I20251024 19:48:04.211318 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a48249f1824a40b087b80a563816d485" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:04.212460 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5473699c1b647f8b86ee43fdb56bdfd" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:04.212633 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d1933bb728845abad5e6b09b24868f0" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:04.212729 1508 raft_consensus.cc:2468] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:04.212957 1509 raft_consensus.cc:3060] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.213790 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4243f11bf2b34b30b10766c19e2c6409
I20251024 19:48:04.213812 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3b4e85f741849b2a6a3e8574396ff88" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:04.213985 1308 leader_election.cc:304] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.214462 1510 raft_consensus.cc:2468] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:04.214789 2141 raft_consensus.cc:493] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.215101 2141 raft_consensus.cc:515] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:04.216764 1447 leader_election.cc:343] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 346383e85cc64b3a8da70a71868ceccc
I20251024 19:48:04.216296 2224 raft_consensus.cc:493] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.217011 2224 raft_consensus.cc:515] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.218372 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet af43ffe097e14c4faffed95e207d0893
I20251024 19:48:04.218519 2217 raft_consensus.cc:2804] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:04.218462 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5473699c1b647f8b86ee43fdb56bdfd" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:04.218627 2217 raft_consensus.cc:493] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:04.218766 2217 raft_consensus.cc:3060] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.219767 1650 raft_consensus.cc:2468] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:04.220305 1371 raft_consensus.cc:2468] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:04.221299 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_82dcb5dcc93042b4a560b867c853f30d" } new_table_name: "default.Table_be0ac003378749608afa5b5a6299c9f0" modify_external_catalogs: true
I20251024 19:48:04.221869 1776 ts_tablet_manager.cc:1929] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:04.221904 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet d38154d3ffd34f4b8d7305b1c23a4fae
I20251024 19:48:04.222309 1774 ts_tablet_manager.cc:1929] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:04.222441 1774 log.cc:1199] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/fe97e31310ae4733ba93de1b62a37842
I20251024 19:48:04.222831 1776 log.cc:1199] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/fe97e31310ae4733ba93de1b62a37842
I20251024 19:48:04.222896 1774 ts_tablet_manager.cc:1950] T fe97e31310ae4733ba93de1b62a37842 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:04.223548 1776 ts_tablet_manager.cc:1950] T fe97e31310ae4733ba93de1b62a37842 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:04.223747 1447 leader_election.cc:304] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:04.224066 2173 raft_consensus.cc:2804] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:04.224263 2173 raft_consensus.cc:697] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:04.224656 2173 consensus_queue.cc:237] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.243516 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet af43ffe097e14c4faffed95e207d0893 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:04.243764 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet d38154d3ffd34f4b8d7305b1c23a4fae with delete_type TABLET_DATA_DELETED (replaced by tablet 68ca2dd438d94bf0b5f1fc69f0a94517) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:04.246551 2224 leader_election.cc:290] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:04.246776 2141 leader_election.cc:290] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:04.255148 1766 raft_consensus.cc:359] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.255404 1766 raft_consensus.cc:385] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.255506 1766 raft_consensus.cc:740] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:04.255779 1766 consensus_queue.cc:260] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.256377 1766 ts_tablet_manager.cc:1434] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.055s user 0.000s sys 0.002s
I20251024 19:48:04.256815 1766 tablet_bootstrap.cc:492] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:04.258539 1444 leader_election.cc:343] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.258901 1444 leader_election.cc:304] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
W20251024 19:48:04.259565 1308 leader_election.cc:343] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.259727 1308 leader_election.cc:304] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.260212 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet d38154d3ffd34f4b8d7305b1c23a4fae with delete_type TABLET_DATA_DELETED (replaced by tablet 68ca2dd438d94bf0b5f1fc69f0a94517) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.260321 2130 raft_consensus.cc:2749] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:04.260740 1188 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 1009ef0615f841dcbee4d58af091193e
I20251024 19:48:04.261020 2224 raft_consensus.cc:2804] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:04.261441 2224 raft_consensus.cc:697] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:04.264961 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 4243f11bf2b34b30b10766c19e2c6409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.266007 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet d1d26e02ad2945b39cf27738ed5b7cf4
I20251024 19:48:04.267520 1190 catalog_manager.cc:6719] ReplaceTablet: tablet d38154d3ffd34f4b8d7305b1c23a4fae deleted and replaced by tablet 68ca2dd438d94bf0b5f1fc69f0a94517
I20251024 19:48:04.269341 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d26e02ad2945b39cf27738ed5b7cf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.274192 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4db17d9aff6542b6b94c09f872a4cceb
I20251024 19:48:04.275491 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5f58bdde786443b5841b7e8d61400808"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:04.277524 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 4db17d9aff6542b6b94c09f872a4cceb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:04.277873 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f58bdde786443b5841b7e8d61400808 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.278147 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8df56cc2917245cca143934fa22702f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.279361 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8df56cc2917245cca143934fa22702f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.279606 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_aa454e7717484495b5d4d77a0951c716"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.280369 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa454e7717484495b5d4d77a0951c716 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.281466 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_998040d863404075b6422120ab4ce290"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:04.282554 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_415e7ca23c72439a9bebab742ea81150"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.283018 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_415e7ca23c72439a9bebab742ea81150 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.285326 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 4db17d9aff6542b6b94c09f872a4cceb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:04.285633 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d26e02ad2945b39cf27738ed5b7cf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:04.286273 1011 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskc9NInd/build/asan/bin/kudu pid 1825
I20251024 19:48:04.286352 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7e15b1eebd3443548f58b749626a8679" } new_table_name: "default.Table_5fb8e365cba24ded8061c38d9c695ff8" modify_external_catalogs: true
I20251024 19:48:04.287106 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "346383e85cc64b3a8da70a71868ceccc" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:04.288190 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5473699c1b647f8b86ee43fdb56bdfd" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:04.288380 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_998040d863404075b6422120ab4ce290 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.288441 2224 consensus_queue.cc:237] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:04.289335 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ad81d80697984ff1834a7d795e0f4b9b
I20251024 19:48:04.289362 2141 raft_consensus.cc:493] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.289554 2141 raft_consensus.cc:515] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.289885 1187 catalog_manager.cc:6719] ReplaceTablet: tablet ad81d80697984ff1834a7d795e0f4b9b deleted and replaced by tablet 7c880a4af94d4041b8c7cf2066002ee3
I20251024 19:48:04.290158 2141 leader_election.cc:290] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:04.291663 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet ad81d80697984ff1834a7d795e0f4b9b with delete_type TABLET_DATA_DELETED (replaced by tablet 7c880a4af94d4041b8c7cf2066002ee3) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.305526 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fc16485e80804cc990dbd6f6aca14502"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.306092 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc16485e80804cc990dbd6f6aca14502 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:04.310709 1444 leader_election.cc:343] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.311180 1444 leader_election.cc:304] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.311496 1509 raft_consensus.cc:2468] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:04.311779 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 350) took 2122 ms. Trace:
I20251024 19:48:04.311947 2224 raft_consensus.cc:2749] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:04.311894 1776 rpcz_store.cc:276] 1024 19:48:02.189301 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.189522 (+ 221us) service_pool.cc:225] Handling call
1024 19:48:03.839210 (+1649688us) ts_tablet_manager.cc:1151] Deleting tablet fe97e31310ae4733ba93de1b62a37842
1024 19:48:03.839218 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:03.951538 (+112320us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.221710 (+270172us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.311713 (+ 90003us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1649448}
W20251024 19:48:04.312242 1444 leader_election.cc:343] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.312520 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 360) took 1905 ms. Trace:
I20251024 19:48:04.312636 1776 rpcz_store.cc:276] 1024 19:48:02.406680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.408104 (+ 1424us) service_pool.cc:225] Handling call
1024 19:48:04.312298 (+1904194us) ts_tablet_manager.cc:1151] Deleting tablet 5d129bdc876e425aabcf1c99997e319a
1024 19:48:04.312313 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.312443 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1880122}
I20251024 19:48:04.313321 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 261) took 3649 ms. Trace:
I20251024 19:48:04.313400 1308 leader_election.cc:304] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.313478 1774 rpcz_store.cc:276] 1024 19:48:00.664337 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:00.664388 (+ 51us) service_pool.cc:225] Handling call
1024 19:48:03.972026 (+3307638us) ts_tablet_manager.cc:1151] Deleting tablet fe97e31310ae4733ba93de1b62a37842
1024 19:48:03.972036 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.133190 (+161154us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.222153 (+ 88963us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.313230 (+ 91077us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3307420}
I20251024 19:48:04.313547 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f346a90a9efe442f82750336e3893deb" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:04.313946 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 268) took 2977 ms. Trace:
I20251024 19:48:04.314054 2130 raft_consensus.cc:2804] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:04.314252 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 361) took 1896 ms. Trace:
I20251024 19:48:04.314428 1775 ts_tablet_manager.cc:1929] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:04.314347 1776 rpcz_store.cc:276] 1024 19:48:02.416617 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.416763 (+ 146us) service_pool.cc:225] Handling call
1024 19:48:04.312908 (+1896145us) ts_tablet_manager.cc:1151] Deleting tablet f3d80885fcc642b8a75da30cd4abecbd
1024 19:48:04.312920 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.313017 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1874160}
W20251024 19:48:04.314579 1444 leader_election.cc:343] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.314735 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 362) took 1895 ms. Trace:
I20251024 19:48:04.314565 1775 log.cc:1199] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/fe97e31310ae4733ba93de1b62a37842
I20251024 19:48:04.314831 1776 rpcz_store.cc:276] 1024 19:48:02.419132 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.483216 (+ 64084us) service_pool.cc:225] Handling call
1024 19:48:04.314586 (+1831370us) ts_tablet_manager.cc:1151] Deleting tablet 129debb55f4a4b7295a8ad88e55ebc83
1024 19:48:04.314597 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.314685 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1817693}
I20251024 19:48:04.315201 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 366) took 1832 ms. Trace:
I20251024 19:48:04.315282 1776 rpcz_store.cc:276] 1024 19:48:02.482770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.482940 (+ 170us) service_pool.cc:225] Handling call
1024 19:48:04.315053 (+1832113us) ts_tablet_manager.cc:1151] Deleting tablet 607ae453221d422e9d07882830c02275
1024 19:48:04.315062 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.315149 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1813868}
I20251024 19:48:04.315631 1775 ts_tablet_manager.cc:1950] T fe97e31310ae4733ba93de1b62a37842 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:04.315661 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 367) took 1831 ms. Trace:
I20251024 19:48:04.315737 1776 rpcz_store.cc:276] 1024 19:48:02.484252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.484535 (+ 283us) service_pool.cc:225] Handling call
1024 19:48:04.315511 (+1830976us) ts_tablet_manager.cc:1151] Deleting tablet d9bf08ccf7054fedb9efcbb00354c71a
1024 19:48:04.315521 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.315614 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1809528}
I20251024 19:48:04.316049 1776 tablet_replica.cc:333] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:04.316020 1774 rpcz_store.cc:276] 1024 19:48:01.336828 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.337217 (+ 389us) service_pool.cc:225] Handling call
1024 19:48:04.313796 (+2976579us) ts_tablet_manager.cc:1151] Deleting tablet 5b96fc2309274048bed7a35e4bf4bae2
1024 19:48:04.313808 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.313905 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2976180}
I20251024 19:48:04.316542 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 269) took 2979 ms. Trace:
I20251024 19:48:04.316663 1776 raft_consensus.cc:2243] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:04.316815 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet fe97e31310ae4733ba93de1b62a37842 (table default.Table_78e6806b2ede4763843892de1ab015e3 [id=1ac4b7fed3f145fc9ef5268266aeb9f1]) successfully deleted
W20251024 19:48:04.317096 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 5d129bdc876e425aabcf1c99997e319a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d129bdc876e425aabcf1c99997e319a already in progress: creating tablet
I20251024 19:48:04.317044 1776 raft_consensus.cc:2272] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:04.317304 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f3d80885fcc642b8a75da30cd4abecbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3d80885fcc642b8a75da30cd4abecbd already in progress: creating tablet
W20251024 19:48:04.317611 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 129debb55f4a4b7295a8ad88e55ebc83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 129debb55f4a4b7295a8ad88e55ebc83 already in progress: creating tablet
W20251024 19:48:04.317831 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 607ae453221d422e9d07882830c02275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607ae453221d422e9d07882830c02275 already in progress: creating tablet
W20251024 19:48:04.318085 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d9bf08ccf7054fedb9efcbb00354c71a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9bf08ccf7054fedb9efcbb00354c71a already in progress: creating tablet
I20251024 19:48:04.318507 2130 raft_consensus.cc:697] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:04.318974 2130 consensus_queue.cc:237] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.321328 1774 rpcz_store.cc:276] 1024 19:48:01.337145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.337650 (+ 505us) service_pool.cc:225] Handling call
1024 19:48:04.316274 (+2978624us) ts_tablet_manager.cc:1151] Deleting tablet 16bc737104ab46b3b071a0760dbc847d
1024 19:48:04.316284 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.316432 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2978407}
I20251024 19:48:04.322121 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 277) took 2876 ms. Trace:
I20251024 19:48:04.322247 1774 rpcz_store.cc:276] 1024 19:48:01.445712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.445758 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:04.321755 (+2875997us) ts_tablet_manager.cc:1151] Deleting tablet 187bc968c2b049028dd25dfb0264c0aa
1024 19:48:04.321770 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.322024 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2875765}
I20251024 19:48:04.322695 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 278) took 2876 ms. Trace:
I20251024 19:48:04.322810 1774 rpcz_store.cc:276] 1024 19:48:01.446125 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.446165 (+ 40us) service_pool.cc:225] Handling call
1024 19:48:04.322545 (+2876380us) ts_tablet_manager.cc:1151] Deleting tablet 59c7dbe9ff2c4869994c4f43d5676c30
1024 19:48:04.322557 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.322660 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2862663}
I20251024 19:48:04.323168 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 279) took 2875 ms. Trace:
I20251024 19:48:04.323276 1774 rpcz_store.cc:276] 1024 19:48:01.447772 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.459982 (+ 12210us) service_pool.cc:225] Handling call
1024 19:48:04.323037 (+2863055us) ts_tablet_manager.cc:1151] Deleting tablet 9489a01434a24d9ca94791252950725c
1024 19:48:04.323048 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.323135 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2862848}
I20251024 19:48:04.323654 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 276) took 2878 ms. Trace:
I20251024 19:48:04.323761 1774 rpcz_store.cc:276] 1024 19:48:01.445527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.446277 (+ 750us) service_pool.cc:225] Handling call
1024 19:48:04.323524 (+2877247us) ts_tablet_manager.cc:1151] Deleting tablet 639836c656f44a819036f4d00e055bd9
1024 19:48:04.323534 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.323624 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2859226}
I20251024 19:48:04.324128 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 291) took 2670 ms. Trace:
I20251024 19:48:04.324245 1774 rpcz_store.cc:276] 1024 19:48:01.653568 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.653670 (+ 102us) service_pool.cc:225] Handling call
1024 19:48:04.323987 (+2670317us) ts_tablet_manager.cc:1151] Deleting tablet 758df87acbdb4adaa83e1ecbc44e7cf1
1024 19:48:04.323997 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.324083 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2668768}
I20251024 19:48:04.324609 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 292) took 2657 ms. Trace:
I20251024 19:48:04.324714 1774 rpcz_store.cc:276] 1024 19:48:01.667110 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.667168 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:04.324478 (+2657310us) ts_tablet_manager.cc:1151] Deleting tablet dc1b6f0a377945089c50012f74d38564
1024 19:48:04.324488 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.324580 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2657084}
I20251024 19:48:04.325058 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 293) took 2633 ms. Trace:
I20251024 19:48:04.325775 1776 ts_tablet_manager.cc:1916] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:04.327101 1766 tablet_bootstrap.cc:654] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:04.329226 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 5b96fc2309274048bed7a35e4bf4bae2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b96fc2309274048bed7a35e4bf4bae2 already in progress: creating tablet
W20251024 19:48:04.329625 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 16bc737104ab46b3b071a0760dbc847d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16bc737104ab46b3b071a0760dbc847d already in progress: creating tablet
W20251024 19:48:04.329875 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 187bc968c2b049028dd25dfb0264c0aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 187bc968c2b049028dd25dfb0264c0aa already in progress: creating tablet
W20251024 19:48:04.330109 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 59c7dbe9ff2c4869994c4f43d5676c30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59c7dbe9ff2c4869994c4f43d5676c30 already in progress: creating tablet
I20251024 19:48:04.330138 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet ad81d80697984ff1834a7d795e0f4b9b with delete_type TABLET_DATA_DELETED (replaced by tablet 7c880a4af94d4041b8c7cf2066002ee3) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:04.330319 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9489a01434a24d9ca94791252950725c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9489a01434a24d9ca94791252950725c already in progress: creating tablet
W20251024 19:48:04.330591 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 639836c656f44a819036f4d00e055bd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 639836c656f44a819036f4d00e055bd9 already in progress: creating tablet
W20251024 19:48:04.330840 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 758df87acbdb4adaa83e1ecbc44e7cf1 already in progress: creating tablet
I20251024 19:48:04.331813 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet 4243f11bf2b34b30b10766c19e2c6409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:04.325233 1774 rpcz_store.cc:276] 1024 19:48:01.691578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.692188 (+ 610us) service_pool.cc:225] Handling call
1024 19:48:04.324942 (+2632754us) ts_tablet_manager.cc:1151] Deleting tablet 62667092d7e244cca1020ec220ea9124
1024 19:48:04.324952 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.325030 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2618661}
I20251024 19:48:04.333936 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 294) took 2616 ms. Trace:
I20251024 19:48:04.335065 1774 rpcz_store.cc:276] 1024 19:48:01.717041 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.739079 (+ 22038us) service_pool.cc:225] Handling call
1024 19:48:04.333521 (+2594442us) ts_tablet_manager.cc:1151] Deleting tablet 6f630fd412ca4c4f8570359c1678e6af
1024 19:48:04.333539 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.333812 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2594208}
I20251024 19:48:04.336124 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f346a90a9efe442f82750336e3893deb" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:04.336544 1372 raft_consensus.cc:2468] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:04.336588 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 295) took 2604 ms. Trace:
I20251024 19:48:04.337702 1774 rpcz_store.cc:276] 1024 19:48:01.732563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.739316 (+ 6753us) service_pool.cc:225] Handling call
1024 19:48:04.336277 (+2596961us) ts_tablet_manager.cc:1151] Deleting tablet d2a68bb1dc6c483388784b9bb19ce065
1024 19:48:04.336291 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.336474 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2596770}
I20251024 19:48:04.338399 1447 leader_election.cc:304] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.339291 2224 raft_consensus.cc:2804] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:04.339501 2224 raft_consensus.cc:493] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:04.339629 2224 raft_consensus.cc:3060] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.339756 2217 raft_consensus.cc:515] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.340291 1370 raft_consensus.cc:2393] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10a9536207e343f188e0ef709081d04e in current term 1: Already voted for candidate 43bec2f03fc849099e254f6335e9d532 in this term.
I20251024 19:48:04.341557 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 296) took 2596 ms. Trace:
I20251024 19:48:04.341902 1774 rpcz_store.cc:276] 1024 19:48:01.745005 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.745055 (+ 50us) service_pool.cc:225] Handling call
1024 19:48:04.341166 (+2596111us) ts_tablet_manager.cc:1151] Deleting tablet fdfa5eea3b674a3389a156d1c72204df
1024 19:48:04.341181 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.341445 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2595865}
I20251024 19:48:04.344210 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet d38154d3ffd34f4b8d7305b1c23a4fae with delete_type TABLET_DATA_DELETED (replaced by tablet 68ca2dd438d94bf0b5f1fc69f0a94517) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.344420 1774 tablet_replica.cc:333] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:04.344800 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d26e02ad2945b39cf27738ed5b7cf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.344820 1774 raft_consensus.cc:2243] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:04.345084 1774 raft_consensus.cc:2272] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:04.345289 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 4243f11bf2b34b30b10766c19e2c6409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.345845 1774 ts_tablet_manager.cc:1916] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:04.348016 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5473699c1b647f8b86ee43fdb56bdfd" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:04.348662 2048 consensus_queue.cc:1048] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:04.348780 1305 leader_election.cc:343] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.349107 2217 leader_election.cc:290] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:04.351066 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5473699c1b647f8b86ee43fdb56bdfd" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:04.351277 1510 raft_consensus.cc:3060] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:04.362149 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 62667092d7e244cca1020ec220ea9124 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62667092d7e244cca1020ec220ea9124 already in progress: creating tablet
W20251024 19:48:04.362463 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 6f630fd412ca4c4f8570359c1678e6af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f630fd412ca4c4f8570359c1678e6af already in progress: creating tablet
W20251024 19:48:04.362705 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet d2a68bb1dc6c483388784b9bb19ce065 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2a68bb1dc6c483388784b9bb19ce065 already in progress: creating tablet
I20251024 19:48:04.363082 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet af43ffe097e14c4faffed95e207d0893 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.366071 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 323) took 2650 ms. Trace:
I20251024 19:48:04.366253 1775 rpcz_store.cc:276] 1024 19:48:01.715364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.715534 (+ 170us) service_pool.cc:225] Handling call
1024 19:48:04.116203 (+2400669us) ts_tablet_manager.cc:1151] Deleting tablet fe97e31310ae4733ba93de1b62a37842
1024 19:48:04.116212 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.214861 (+ 98649us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.314216 (+ 99355us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.366008 (+ 51792us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2400430}
I20251024 19:48:04.366883 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 324) took 2649 ms. Trace:
I20251024 19:48:04.367158 1775 rpcz_store.cc:276] 1024 19:48:01.717300 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.717384 (+ 84us) service_pool.cc:225] Handling call
1024 19:48:04.366696 (+2649312us) ts_tablet_manager.cc:1151] Deleting tablet 6f630fd412ca4c4f8570359c1678e6af
1024 19:48:04.366711 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.366838 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2649043}
I20251024 19:48:04.367672 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 325) took 2634 ms. Trace:
I20251024 19:48:04.367815 1775 rpcz_store.cc:276] 1024 19:48:01.732796 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.732895 (+ 99us) service_pool.cc:225] Handling call
1024 19:48:04.367484 (+2634589us) ts_tablet_manager.cc:1151] Deleting tablet d2a68bb1dc6c483388784b9bb19ce065
1024 19:48:04.367495 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.367624 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2634279}
I20251024 19:48:04.368237 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 326) took 2624 ms. Trace:
I20251024 19:48:04.368352 1775 rpcz_store.cc:276] 1024 19:48:01.744031 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.744088 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:04.368085 (+2623997us) ts_tablet_manager.cc:1151] Deleting tablet fdfa5eea3b674a3389a156d1c72204df
1024 19:48:04.368095 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.368200 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2623757}
I20251024 19:48:04.368690 1775 tablet_replica.cc:333] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:04.368957 1775 raft_consensus.cc:2243] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:04.369196 1775 raft_consensus.cc:2272] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:04.370270 1775 ts_tablet_manager.cc:1916] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:04.370795 1768 tablet_bootstrap.cc:492] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:04.370983 1768 ts_tablet_manager.cc:1403] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.457s user 0.102s sys 0.132s
W20251024 19:48:04.371578 1767 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.012s sys 0.039s
W20251024 19:48:04.371709 1767 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.512s user 0.012s sys 0.039s
I20251024 19:48:04.372151 1768 raft_consensus.cc:359] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:04.372424 1768 raft_consensus.cc:385] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.372509 1768 raft_consensus.cc:740] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
W20251024 19:48:04.372908 1745 thread.cc:641] leader-initialization [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.019s sys 0.040s
I20251024 19:48:04.372817 1768 consensus_queue.cc:260] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:04.373030 1745 thread.cc:608] leader-initialization [worker] (thread pool) Time spent starting thread: real 0.565s user 0.019s sys 0.040s
I20251024 19:48:04.373379 1768 ts_tablet_manager.cc:1434] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:04.376554 1766 tablet_bootstrap.cc:492] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:04.376703 1766 ts_tablet_manager.cc:1403] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.120s user 0.004s sys 0.001s
I20251024 19:48:04.377753 1766 raft_consensus.cc:359] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.378067 1766 raft_consensus.cc:385] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.378201 1766 raft_consensus.cc:740] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:04.378434 1766 consensus_queue.cc:260] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:04.362998 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet fdfa5eea3b674a3389a156d1c72204df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdfa5eea3b674a3389a156d1c72204df already in progress: creating tablet
I20251024 19:48:04.391695 1447 leader_election.cc:304] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:04.392499 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 6f630fd412ca4c4f8570359c1678e6af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f630fd412ca4c4f8570359c1678e6af already in progress: creating tablet
W20251024 19:48:04.392793 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet d2a68bb1dc6c483388784b9bb19ce065 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2a68bb1dc6c483388784b9bb19ce065 already in progress: creating tablet
W20251024 19:48:04.393052 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet fdfa5eea3b674a3389a156d1c72204df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdfa5eea3b674a3389a156d1c72204df already in progress: creating tablet
I20251024 19:48:04.400746 1767 tablet_bootstrap.cc:492] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:04.401124 1767 ts_tablet_manager.cc:1403] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.545s user 0.013s sys 0.041s
I20251024 19:48:04.402581 1767 raft_consensus.cc:359] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.402890 1767 raft_consensus.cc:385] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.402997 1767 raft_consensus.cc:740] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:04.403304 1767 consensus_queue.cc:260] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.406507 2013 generic_service.cc:196] Checking for leaks (request via RPC)
I20251024 19:48:04.409904 1768 tablet_bootstrap.cc:492] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:04.412607 1768 tablet_bootstrap.cc:654] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:04.413671 1766 ts_tablet_manager.cc:1434] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:04.414183 1766 tablet_bootstrap.cc:492] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:04.414127 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 4db17d9aff6542b6b94c09f872a4cceb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.414362 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet ad81d80697984ff1834a7d795e0f4b9b with delete_type TABLET_DATA_DELETED (replaced by tablet 7c880a4af94d4041b8c7cf2066002ee3) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.416214 1766 tablet_bootstrap.cc:654] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:04.418325 2224 raft_consensus.cc:515] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.419392 2224 leader_election.cc:290] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:04.420080 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f346a90a9efe442f82750336e3893deb" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
W20251024 19:48:04.420881 1444 leader_election.cc:343] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.426028 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet af43ffe097e14c4faffed95e207d0893 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.430666 2224 raft_consensus.cc:493] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.430920 2224 raft_consensus.cc:515] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.431465 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f346a90a9efe442f82750336e3893deb" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:04.431737 1370 raft_consensus.cc:3060] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.432041 2224 leader_election.cc:290] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:04.432991 2224 consensus_queue.cc:1048] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:04.435766 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50d919a0b3d74e9a97d136a84d50e24b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:04.436089 1372 raft_consensus.cc:2468] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:04.436553 1447 leader_election.cc:304] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:04.436877 2224 raft_consensus.cc:2804] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:04.437009 2224 raft_consensus.cc:493] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:04.437218 2224 raft_consensus.cc:3060] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.440155 1768 tablet_bootstrap.cc:492] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:04.440426 1768 ts_tablet_manager.cc:1403] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.001s
I20251024 19:48:04.442010 1768 raft_consensus.cc:359] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:04.442272 1768 raft_consensus.cc:385] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.442502 1768 raft_consensus.cc:740] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:04.443562 1766 tablet_bootstrap.cc:492] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:04.444015 1766 ts_tablet_manager.cc:1403] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.001s
I20251024 19:48:04.445024 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50d919a0b3d74e9a97d136a84d50e24b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:04.445943 1444 leader_election.cc:343] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.449914 1768 consensus_queue.cc:260] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:04.451082 1768 ts_tablet_manager.cc:1434] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20251024 19:48:04.451665 1768 tablet_bootstrap.cc:492] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:04.460500 2146 consensus_queue.cc:1048] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:04.460489 1766 raft_consensus.cc:359] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.460909 1766 raft_consensus.cc:385] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.461061 1766 raft_consensus.cc:740] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:04.461469 1766 consensus_queue.cc:260] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.463172 1766 ts_tablet_manager.cc:1434] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20251024 19:48:04.463532 1766 tablet_bootstrap.cc:492] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:04.465523 1766 tablet_bootstrap.cc:654] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:04.468207 1767 ts_tablet_manager.cc:1434] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.059s user 0.000s sys 0.003s
I20251024 19:48:04.468690 1767 tablet_bootstrap.cc:492] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:04.476483 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d95afcc3eb2e481182edd9cffc3b5b50" } new_table_name: "default.Table_0fa2426ca885458797e98bedc44dec04" modify_external_catalogs: true
I20251024 19:48:04.477698 1768 tablet_bootstrap.cc:654] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:04.478129 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2622ff0be0624668b5534e85df77e487"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.478694 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2622ff0be0624668b5534e85df77e487 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.480086 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_511ba1e13b8f43e299835a4c30063587" } new_table_name: "default.Table_f11b271966af455798227788699910b6" modify_external_catalogs: true
I20251024 19:48:04.481436 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_df7191b6e1d94c5fae650a13660bf68b" } new_table_name: "default.Table_3b9aa7dd64334765afe410d802b10292" modify_external_catalogs: true
I20251024 19:48:04.482654 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet d38154d3ffd34f4b8d7305b1c23a4fae from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:04.483297 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 774a2de2e3764c54b72acfb952ed3e11 consensus state has no leader
I20251024 19:48:04.483562 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet f83b9290f00145a780318a43249eec12 from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:04.483947 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b632acc1348e49f8a1eea019c007780c consensus state has no leader
I20251024 19:48:04.484148 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet 129debb55f4a4b7295a8ad88e55ebc83 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:04.484790 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f37eb464f5144d1a8135a5f1a44cee4f" } modify_external_catalogs: true
I20251024 19:48:04.484958 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f37eb464f5144d1a8135a5f1a44cee4f" } modify_external_catalogs: true
I20251024 19:48:04.486033 2130 raft_consensus.cc:493] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.486183 2130 raft_consensus.cc:515] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.486413 1766 tablet_bootstrap.cc:492] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:04.486559 1766 ts_tablet_manager.cc:1403] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.003s
I20251024 19:48:04.486877 2130 leader_election.cc:290] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:04.487380 1766 raft_consensus.cc:359] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.487545 1766 raft_consensus.cc:385] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.487629 1766 raft_consensus.cc:740] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:04.487902 1766 consensus_queue.cc:260] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.488782 1766 ts_tablet_manager.cc:1434] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251024 19:48:04.489137 1766 tablet_bootstrap.cc:492] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:04.491261 1766 tablet_bootstrap.cc:654] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:04.491652 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "857eed60ba2a4b309e86d88a0f093762" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:04.492439 1305 leader_election.cc:343] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.496555 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "857eed60ba2a4b309e86d88a0f093762" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:04.496768 1509 raft_consensus.cc:2468] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:04.497145 1308 leader_election.cc:304] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.497601 2130 raft_consensus.cc:2804] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:04.497725 2130 raft_consensus.cc:493] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:04.497825 2130 raft_consensus.cc:3060] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.501116 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6b41a041a50a4f2ca796ea21033f1ab9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.501893 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b41a041a50a4f2ca796ea21033f1ab9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.509048 2217 raft_consensus.cc:493] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.509301 2217 raft_consensus.cc:515] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:04.510135 2217 leader_election.cc:290] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:04.513044 1370 raft_consensus.cc:2468] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:04.513625 1776 ts_tablet_manager.cc:1929] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:04.513850 1776 log.cc:1199] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/f05ad092936b4bafa3ae795904c2df78
I20251024 19:48:04.514611 1776 ts_tablet_manager.cc:1950] T f05ad092936b4bafa3ae795904c2df78 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:04.518625 1774 ts_tablet_manager.cc:1929] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:04.518795 1774 log.cc:1199] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/e3b4e85f741849b2a6a3e8574396ff88
I20251024 19:48:04.519233 1774 ts_tablet_manager.cc:1950] T e3b4e85f741849b2a6a3e8574396ff88 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:04.519786 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc1b6f0a377945089c50012f74d38564" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:04.520682 2130 raft_consensus.cc:515] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.521449 2130 leader_election.cc:290] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:04.521456 1305 leader_election.cc:343] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.522382 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "857eed60ba2a4b309e86d88a0f093762" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
W20251024 19:48:04.522943 1305 leader_election.cc:343] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.524030 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 368) took 2034 ms. Trace:
I20251024 19:48:04.524107 1510 raft_consensus.cc:2468] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:04.524197 1776 rpcz_store.cc:276] 1024 19:48:02.489088 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.493133 (+ 4045us) service_pool.cc:225] Handling call
1024 19:48:04.315968 (+1822835us) ts_tablet_manager.cc:1151] Deleting tablet f05ad092936b4bafa3ae795904c2df78
1024 19:48:04.315979 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.409992 (+ 94013us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.513370 (+103378us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.523981 (+ 10611us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1804382}
I20251024 19:48:04.524616 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 370) took 1985 ms. Trace:
I20251024 19:48:04.524686 2141 raft_consensus.cc:2749] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:04.524717 1776 rpcz_store.cc:276] 1024 19:48:02.538975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.539027 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:04.524491 (+1985464us) ts_tablet_manager.cc:1151] Deleting tablet 80c58e2439dc490c824b5f0023b5434b
1024 19:48:04.524501 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.524588 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1978650}
I20251024 19:48:04.525025 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 379) took 1921 ms. Trace:
I20251024 19:48:04.525110 1776 rpcz_store.cc:276] 1024 19:48:02.603245 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.603992 (+ 747us) service_pool.cc:225] Handling call
1024 19:48:04.524923 (+1920931us) ts_tablet_manager.cc:1151] Deleting tablet b79801594e95434cbb8fe7b75b14602e
1024 19:48:04.524931 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.524999 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1898832}
I20251024 19:48:04.525427 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 388) took 1794 ms. Trace:
I20251024 19:48:04.525508 1776 rpcz_store.cc:276] 1024 19:48:02.731153 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.731281 (+ 128us) service_pool.cc:225] Handling call
1024 19:48:04.525288 (+1794007us) ts_tablet_manager.cc:1151] Deleting tablet 4a69c0918d2e4eed9d91b1913ec2c21e
1024 19:48:04.525295 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.525383 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1781580}
I20251024 19:48:04.525631 1308 leader_election.cc:304] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.525830 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 389) took 1785 ms. Trace:
I20251024 19:48:04.525920 1776 rpcz_store.cc:276] 1024 19:48:02.740078 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.740210 (+ 132us) service_pool.cc:225] Handling call
1024 19:48:04.525715 (+1785505us) ts_tablet_manager.cc:1151] Deleting tablet 2e7b3288182744969e217d4f174fe4ea
1024 19:48:04.525724 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.525802 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1776804}
I20251024 19:48:04.526126 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "857eed60ba2a4b309e86d88a0f093762" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:04.526196 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 397) took 1489 ms. Trace:
I20251024 19:48:04.526278 1776 rpcz_store.cc:276] 1024 19:48:03.036772 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.036836 (+ 64us) service_pool.cc:225] Handling call
1024 19:48:04.526086 (+1489250us) ts_tablet_manager.cc:1151] Deleting tablet d76410afd7ba4c73836f4791ef739056
1024 19:48:04.526094 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.526170 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1488953}
I20251024 19:48:04.526544 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 396) took 1489 ms. Trace:
I20251024 19:48:04.526685 1776 rpcz_store.cc:276] 1024 19:48:03.036614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.037182 (+ 568us) service_pool.cc:225] Handling call
1024 19:48:04.526444 (+1489262us) ts_tablet_manager.cc:1151] Deleting tablet dc14b0cdb7f342ac956a7857ab3f35d7
1024 19:48:04.526451 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.526512 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1489029}
I20251024 19:48:04.526916 2217 raft_consensus.cc:2804] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:04.527141 2217 raft_consensus.cc:697] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:04.527314 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet f05ad092936b4bafa3ae795904c2df78 (table default.Table_99584fc98c7e4f0f853914cc3dda2b12 [id=0704c52aae2c421789a94e4391b9ed2a]) successfully deleted
I20251024 19:48:04.527645 2217 consensus_queue.cc:237] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:04.527710 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 80c58e2439dc490c824b5f0023b5434b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80c58e2439dc490c824b5f0023b5434b already in progress: creating tablet
W20251024 19:48:04.528071 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b79801594e95434cbb8fe7b75b14602e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b79801594e95434cbb8fe7b75b14602e already in progress: creating tablet
W20251024 19:48:04.528326 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4a69c0918d2e4eed9d91b1913ec2c21e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a69c0918d2e4eed9d91b1913ec2c21e already in progress: creating tablet
W20251024 19:48:04.528532 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2e7b3288182744969e217d4f174fe4ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e7b3288182744969e217d4f174fe4ea already in progress: creating tablet
W20251024 19:48:04.528921 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d76410afd7ba4c73836f4791ef739056 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d76410afd7ba4c73836f4791ef739056 already in progress: creating tablet
I20251024 19:48:04.529076 1510 raft_consensus.cc:3060] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:04.529204 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet dc14b0cdb7f342ac956a7857ab3f35d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc14b0cdb7f342ac956a7857ab3f35d7 already in progress: creating tablet
I20251024 19:48:04.530292 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 395) took 1493 ms. Trace:
I20251024 19:48:04.530551 1776 rpcz_store.cc:276] 1024 19:48:03.036536 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.036833 (+ 297us) service_pool.cc:225] Handling call
1024 19:48:04.530038 (+1493205us) ts_tablet_manager.cc:1151] Deleting tablet bc31e6c25ad14947af20abd766f12e78
1024 19:48:04.530049 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.530204 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1491950}
W20251024 19:48:04.531126 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet bc31e6c25ad14947af20abd766f12e78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc31e6c25ad14947af20abd766f12e78 already in progress: creating tablet
I20251024 19:48:04.531646 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 403) took 1390 ms. Trace:
I20251024 19:48:04.531898 1776 rpcz_store.cc:276] 1024 19:48:03.141009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.141054 (+ 45us) service_pool.cc:225] Handling call
1024 19:48:04.531391 (+1390337us) ts_tablet_manager.cc:1151] Deleting tablet 9a1031d785b646b095ece9be666d2a77
1024 19:48:04.531400 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.531551 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1388608}
W20251024 19:48:04.532397 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9a1031d785b646b095ece9be666d2a77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a1031d785b646b095ece9be666d2a77 already in progress: creating tablet
I20251024 19:48:04.532819 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 402) took 1391 ms. Trace:
I20251024 19:48:04.532958 1776 rpcz_store.cc:276] 1024 19:48:03.140888 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.141230 (+ 342us) service_pool.cc:225] Handling call
1024 19:48:04.532670 (+1391440us) ts_tablet_manager.cc:1151] Deleting tablet d840888183ec4ef7bb245363c5bac322
1024 19:48:04.532680 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.532787 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1381485}
W20251024 19:48:04.533247 1179 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 46c76d86a68c40e18cc176b4776db266 (table default.Table_dbe9a03f95bf479490b2015a0c6afe20 [id=51716cd9fedf415bb03a962bb4235b42], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 46c76d86a68c40e18cc176b4776db266 consensus state has no leader
I20251024 19:48:04.533459 2141 consensus_queue.cc:1048] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
W20251024 19:48:04.533771 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d840888183ec4ef7bb245363c5bac322 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d840888183ec4ef7bb245363c5bac322 already in progress: creating tablet
I20251024 19:48:04.534088 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 401) took 1393 ms. Trace:
I20251024 19:48:04.534188 1776 rpcz_store.cc:276] 1024 19:48:03.140755 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.141328 (+ 573us) service_pool.cc:225] Handling call
1024 19:48:04.533973 (+1392645us) ts_tablet_manager.cc:1151] Deleting tablet 7d1933bb728845abad5e6b09b24868f0
1024 19:48:04.533982 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.534060 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1377670}
W20251024 19:48:04.534549 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7d1933bb728845abad5e6b09b24868f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d1933bb728845abad5e6b09b24868f0 already in progress: creating tablet
I20251024 19:48:04.534823 1776 tablet_replica.cc:333] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:04.535070 1776 raft_consensus.cc:2243] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:04.535419 1776 raft_consensus.cc:2272] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:04.536283 1776 ts_tablet_manager.cc:1916] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:04.524008 1767 tablet_bootstrap.cc:654] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:04.545301 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc1b6f0a377945089c50012f74d38564" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:04.546985 1308 leader_election.cc:343] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.547189 1308 leader_election.cc:304] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.547832 2217 raft_consensus.cc:2749] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:04.549213 1768 tablet_bootstrap.cc:492] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:04.549597 1768 ts_tablet_manager.cc:1403] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.098s user 0.001s sys 0.009s
I20251024 19:48:04.551586 1768 raft_consensus.cc:359] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.552392 1447 leader_election.cc:304] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.552860 2141 raft_consensus.cc:2804] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:04.553031 2141 raft_consensus.cc:697] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:04.553375 2141 consensus_queue.cc:237] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.558036 1767 tablet_bootstrap.cc:492] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:04.558233 1767 ts_tablet_manager.cc:1403] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.090s user 0.004s sys 0.000s
I20251024 19:48:04.559103 1767 raft_consensus.cc:359] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.559314 1767 raft_consensus.cc:385] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.559420 1767 raft_consensus.cc:740] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:04.559660 1767 consensus_queue.cc:260] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.560343 1767 ts_tablet_manager.cc:1434] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251024 19:48:04.560688 1767 tablet_bootstrap.cc:492] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:04.552523 1766 tablet_bootstrap.cc:492] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:04.560822 1768 raft_consensus.cc:385] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.560945 1768 raft_consensus.cc:740] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:04.560947 1766 ts_tablet_manager.cc:1403] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.002s
I20251024 19:48:04.562000 1766 raft_consensus.cc:359] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.562300 1766 raft_consensus.cc:385] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.562394 1766 raft_consensus.cc:740] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:04.562574 1767 tablet_bootstrap.cc:654] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:04.564205 1768 consensus_queue.cc:260] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.564340 1766 consensus_queue.cc:260] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.564867 1768 ts_tablet_manager.cc:1434] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251024 19:48:04.564970 1766 ts_tablet_manager.cc:1434] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:04.565358 1766 tablet_bootstrap.cc:492] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:04.567603 1766 tablet_bootstrap.cc:654] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:04.568792 1768 tablet_bootstrap.cc:492] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:04.570816 2141 raft_consensus.cc:493] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.571025 2141 raft_consensus.cc:515] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.572070 2141 leader_election.cc:290] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:04.572480 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913534ab88dd459e9cac2faebb5492a1" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:04.573202 1444 leader_election.cc:343] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.573434 2217 raft_consensus.cc:493] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.573762 2217 raft_consensus.cc:515] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.575316 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81f8393f78204860bd854f4872099387" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:04.576800 2217 leader_election.cc:290] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:04.577136 1768 tablet_bootstrap.cc:654] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:04.578130 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81f8393f78204860bd854f4872099387" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:04.578712 1767 tablet_bootstrap.cc:492] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:04.578892 1767 ts_tablet_manager.cc:1403] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
W20251024 19:48:04.581725 1308 leader_election.cc:343] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.581614 1767 raft_consensus.cc:359] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.581986 1767 raft_consensus.cc:385] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.582113 1767 raft_consensus.cc:740] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:04.582374 1767 consensus_queue.cc:260] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.583019 1767 ts_tablet_manager.cc:1434] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251024 19:48:04.583340 1767 tablet_bootstrap.cc:492] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:48:04.583878 1305 leader_election.cc:343] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.584003 1305 leader_election.cc:304] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.584426 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913534ab88dd459e9cac2faebb5492a1" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:04.584614 2217 raft_consensus.cc:2749] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:04.584760 1370 raft_consensus.cc:2468] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:04.585050 1767 tablet_bootstrap.cc:654] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:04.585394 1447 leader_election.cc:304] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.586180 2141 raft_consensus.cc:2804] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:04.586447 2141 raft_consensus.cc:493] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:04.586688 2141 raft_consensus.cc:3060] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.587639 1766 tablet_bootstrap.cc:492] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:04.587857 1766 ts_tablet_manager.cc:1403] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20251024 19:48:04.588840 1766 raft_consensus.cc:359] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.589033 1766 raft_consensus.cc:385] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.589141 1766 raft_consensus.cc:740] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:04.589380 1766 consensus_queue.cc:260] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.590051 1766 ts_tablet_manager.cc:1434] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251024 19:48:04.590370 1766 tablet_bootstrap.cc:492] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:04.593629 1766 tablet_bootstrap.cc:654] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:04.615571 1510 raft_consensus.cc:2468] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:04.616212 1775 ts_tablet_manager.cc:1929] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:04.616370 1775 log.cc:1199] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/f05ad092936b4bafa3ae795904c2df78
I20251024 19:48:04.618235 1775 ts_tablet_manager.cc:1950] T f05ad092936b4bafa3ae795904c2df78 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:04.617974 1187 catalog_manager.cc:5649] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:04.618498 1187 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet c04082bd1b444b468c177bef239701c6 (table default.Table_24200cd6f19d460cae963afb9b7f8724 [id=a2789faf4abe4c09b0a7515d941831a6]). Expected version 2 got 0
I20251024 19:48:04.619949 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_63c72c09261e44538e2cb5586550b43f" } modify_external_catalogs: true
I20251024 19:48:04.620172 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_63c72c09261e44538e2cb5586550b43f" } modify_external_catalogs: true
I20251024 19:48:04.621029 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_747b358db68e43159e1616089b57313a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:04.622151 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 305) took 2571 ms. Trace:
I20251024 19:48:04.622375 1774 rpcz_store.cc:276] 1024 19:48:02.044546 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.045443 (+ 897us) service_pool.cc:225] Handling call
1024 19:48:04.344238 (+2298795us) ts_tablet_manager.cc:1151] Deleting tablet e3b4e85f741849b2a6a3e8574396ff88
1024 19:48:04.344252 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.409992 (+ 65740us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.518490 (+108498us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.615257 (+ 96767us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2288498}
W20251024 19:48:04.622833 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15de0d8aa9e545c09233186bbe361b97 consensus state has no leader
I20251024 19:48:04.624361 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 304) took 2580 ms. Trace:
I20251024 19:48:04.625784 1308 leader_election.cc:304] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.626648 2217 raft_consensus.cc:2804] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:04.626902 2217 raft_consensus.cc:697] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:04.627966 2217 consensus_queue.cc:237] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.624564 1774 rpcz_store.cc:276] 1024 19:48:02.043149 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.047912 (+ 4763us) service_pool.cc:225] Handling call
1024 19:48:04.622824 (+2574912us) ts_tablet_manager.cc:1151] Deleting tablet 1edc55caab904722bf19a1c71df601cf
1024 19:48:04.622840 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.623100 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2559881}
I20251024 19:48:04.630599 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 306) took 2568 ms. Trace:
I20251024 19:48:04.630715 1774 rpcz_store.cc:276] 1024 19:48:02.062203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.062259 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:04.630358 (+2568099us) ts_tablet_manager.cc:1151] Deleting tablet 3e6db58fac5e4af9b27bde6c223aa845
1024 19:48:04.630373 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.630531 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2564559}
I20251024 19:48:04.631095 1774 tablet_replica.cc:333] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:04.631371 1774 raft_consensus.cc:2243] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:04.631592 1774 raft_consensus.cc:2272] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:04.632244 1774 ts_tablet_manager.cc:1916] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:04.633757 2130 raft_consensus.cc:493] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.634549 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c357a2a81064b3891afda6d252898b9 with delete_type TABLET_DATA_DELETED (replaced by tablet d38154d3ffd34f4b8d7305b1c23a4fae) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.633931 2130 raft_consensus.cc:515] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.637892 2130 leader_election.cc:290] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:04.639272 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ee03ede02c4bdab706fde910978332" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:04.639994 1308 leader_election.cc:343] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.641335 1611 tablet_service.cc:1505] Processing CreateTablet for tablet be21ca442dee4204a8b15f5a2dd9530b (DEFAULT_TABLE table=default.Table_ea0c3c7222394ffda2136fe114e6760b [id=6c7880abb36140c6afb52f0b38558770]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.642601 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_25adc9559f2b43e2a4adc92dfc0c7807" } modify_external_catalogs: true
I20251024 19:48:04.647140 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ee03ede02c4bdab706fde910978332" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:04.649868 1305 leader_election.cc:343] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.650040 1305 leader_election.cc:304] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.652225 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 24cb7ec9294d4f438a042451f669b98f (DEFAULT_TABLE table=default.Table_7af095414d2e40c08193c763d497a755 [id=3598c839c14a4f53b3227e4aca2d9350]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.652331 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 1295c99836d14a1cb0002e6c6aaee1eb (DEFAULT_TABLE table=default.Table_631ebe0a47914d38854b5d13f0ea4af4 [id=48cb957953844690a7e234fae84e4768]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.652969 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 3b8e8dbca79f4d72a2de51f105b209e4 (DEFAULT_TABLE table=default.Table_5834ffa88ad9439a9052f00f4859d799 [id=a00aaf99443a4765be2a095c601bc361]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.653867 2130 raft_consensus.cc:2749] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:04.654577 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_546f0aaae7694ef9807723a9ecc39602" } modify_external_catalogs: true
I20251024 19:48:04.655967 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_24200cd6f19d460cae963afb9b7f8724" } modify_external_catalogs: true
I20251024 19:48:04.657135 2246 raft_consensus.cc:493] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.657281 2246 raft_consensus.cc:515] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.658028 2246 leader_election.cc:290] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:04.658969 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c357a2a81064b3891afda6d252898b9" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:04.659242 1510 raft_consensus.cc:2468] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:04.659767 1586 leader_election.cc:304] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:04.660082 2246 raft_consensus.cc:2804] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:04.660188 2246 raft_consensus.cc:493] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:04.660279 2246 raft_consensus.cc:3060] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.661114 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 163ac050c8574615aef5061f563cfc0a
I20251024 19:48:04.664860 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 163ac050c8574615aef5061f563cfc0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.665205 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c357a2a81064b3891afda6d252898b9" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:04.665457 1370 raft_consensus.cc:2468] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:04.669310 2130 raft_consensus.cc:493] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.669440 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet e3b4e85f741849b2a6a3e8574396ff88 (table default.Table_b905a1285caa407e9fb08e81639c9bc8 [id=1da47e6010f548d1b3d8c538150a8a8e]) successfully deleted
W20251024 19:48:04.669628 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 45342.8us
I20251024 19:48:04.669621 2130 raft_consensus.cc:515] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:04.670539 2130 leader_election.cc:290] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:04.670658 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 1edc55caab904722bf19a1c71df601cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1edc55caab904722bf19a1c71df601cf already in progress: creating tablet
W20251024 19:48:04.670872 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3e6db58fac5e4af9b27bde6c223aa845 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e6db58fac5e4af9b27bde6c223aa845 already in progress: creating tablet
W20251024 19:48:04.671032 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 452f77091adc4cf09de2231f2f6d845b consensus state has no leader
W20251024 19:48:04.671597 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64499c8734d34c66b0b52a2cd3ef0673 consensus state has no leader
I20251024 19:48:04.671936 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 24cb7ec9294d4f438a042451f669b98f (DEFAULT_TABLE table=default.Table_7af095414d2e40c08193c763d497a755 [id=3598c839c14a4f53b3227e4aca2d9350]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.672381 1336 tablet_service.cc:1505] Processing CreateTablet for tablet be21ca442dee4204a8b15f5a2dd9530b (DEFAULT_TABLE table=default.Table_ea0c3c7222394ffda2136fe114e6760b [id=6c7880abb36140c6afb52f0b38558770]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.672515 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 163ac050c8574615aef5061f563cfc0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:04.673134 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 1295c99836d14a1cb0002e6c6aaee1eb (DEFAULT_TABLE table=default.Table_631ebe0a47914d38854b5d13f0ea4af4 [id=48cb957953844690a7e234fae84e4768]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.673944 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 3b8e8dbca79f4d72a2de51f105b209e4 (DEFAULT_TABLE table=default.Table_5834ffa88ad9439a9052f00f4859d799 [id=a00aaf99443a4765be2a095c601bc361]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.675283 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7ab38232d57d40fea8917ef3a67df1ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.675936 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ab38232d57d40fea8917ef3a67df1ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.676180 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 25ced70ff40d4149bb92e8b3767ec38f (DEFAULT_TABLE table=default.Table_b9d913c84cec4d508db7fe4c3dc63e27 [id=eacc2d37d57f4d588101cb545db90c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.677166 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_25adc9559f2b43e2a4adc92dfc0c7807" } modify_external_catalogs: true
I20251024 19:48:04.677596 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d04a9b54bd6446d8deb64292cef672d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:04.677958 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f83b9290f00145a780318a43249eec12
I20251024 19:48:04.678359 1184 catalog_manager.cc:6719] ReplaceTablet: tablet f83b9290f00145a780318a43249eec12 deleted and replaced by tablet 41176885447e42409ee4363bb4d903f2
W20251024 19:48:04.678399 1308 leader_election.cc:343] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.678968 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c357a2a81064b3891afda6d252898b9 with delete_type TABLET_DATA_DELETED (replaced by tablet d38154d3ffd34f4b8d7305b1c23a4fae) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.679562 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 2496f103436047dea5dabb577651249f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.679797 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 346383e85cc64b3a8da70a71868ceccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.680163 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet f83b9290f00145a780318a43249eec12 with delete_type TABLET_DATA_DELETED (replaced by tablet 41176885447e42409ee4363bb4d903f2) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.680202 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 163ac050c8574615aef5061f563cfc0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.680398 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 1295c99836d14a1cb0002e6c6aaee1eb (DEFAULT_TABLE table=default.Table_631ebe0a47914d38854b5d13f0ea4af4 [id=48cb957953844690a7e234fae84e4768]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.680842 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1295c99836d14a1cb0002e6c6aaee1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.680955 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d04a9b54bd6446d8deb64292cef672d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:04.682163 1305 leader_election.cc:343] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.682333 1305 leader_election.cc:304] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.682955 2130 raft_consensus.cc:2749] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:04.685123 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 24cb7ec9294d4f438a042451f669b98f (DEFAULT_TABLE table=default.Table_7af095414d2e40c08193c763d497a755 [id=3598c839c14a4f53b3227e4aca2d9350]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.685537 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cb7ec9294d4f438a042451f669b98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.685864 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 3b8e8dbca79f4d72a2de51f105b209e4 (DEFAULT_TABLE table=default.Table_5834ffa88ad9439a9052f00f4859d799 [id=a00aaf99443a4765be2a095c601bc361]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.687857 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8e8dbca79f4d72a2de51f105b209e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.686324 1484 tablet_service.cc:1505] Processing CreateTablet for tablet be21ca442dee4204a8b15f5a2dd9530b (DEFAULT_TABLE table=default.Table_ea0c3c7222394ffda2136fe114e6760b [id=6c7880abb36140c6afb52f0b38558770]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.693159 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be21ca442dee4204a8b15f5a2dd9530b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.695983 1768 tablet_bootstrap.cc:492] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:04.696179 1768 ts_tablet_manager.cc:1403] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.131s user 0.002s sys 0.005s
I20251024 19:48:04.697214 1768 raft_consensus.cc:359] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.697436 1768 raft_consensus.cc:385] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.697541 1768 raft_consensus.cc:740] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:04.698118 1768 consensus_queue.cc:260] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.698823 1768 ts_tablet_manager.cc:1434] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:04.699172 1768 tablet_bootstrap.cc:492] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:04.699801 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet f83b9290f00145a780318a43249eec12 with delete_type TABLET_DATA_DELETED (replaced by tablet 41176885447e42409ee4363bb4d903f2) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:04.702080 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cb7ec9294d4f438a042451f669b98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.702342 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1295c99836d14a1cb0002e6c6aaee1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.702790 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be21ca442dee4204a8b15f5a2dd9530b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.702858 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet f83b9290f00145a780318a43249eec12 with delete_type TABLET_DATA_DELETED (replaced by tablet 41176885447e42409ee4363bb4d903f2) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.703096 2020 consensus_queue.cc:1048] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:04.703482 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8e8dbca79f4d72a2de51f105b209e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.705544 1768 tablet_bootstrap.cc:654] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:04.707576 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_24a6bf47782a48f98a76f6bcda40352e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.708364 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24a6bf47782a48f98a76f6bcda40352e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.708601 2020 raft_consensus.cc:493] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.708894 2020 raft_consensus.cc:515] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.709908 2020 leader_election.cc:290] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:04.710346 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c8e3fdadfb964ad6b614c60e5d2695c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.710954 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8e3fdadfb964ad6b614c60e5d2695c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.711038 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81f8393f78204860bd854f4872099387" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:04.711678 1444 leader_election.cc:343] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.713454 2224 raft_consensus.cc:515] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.714375 2224 leader_election.cc:290] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:04.718143 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_546f0aaae7694ef9807723a9ecc39602" } modify_external_catalogs: true
I20251024 19:48:04.719622 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_24200cd6f19d460cae963afb9b7f8724" } modify_external_catalogs: true
I20251024 19:48:04.713117 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 335) took 2767 ms. Trace:
I20251024 19:48:04.721105 1775 rpcz_store.cc:276] 1024 19:48:01.945758 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.946739 (+ 981us) service_pool.cc:225] Handling call
1024 19:48:04.368599 (+2421860us) ts_tablet_manager.cc:1151] Deleting tablet f05ad092936b4bafa3ae795904c2df78
1024 19:48:04.368610 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.514988 (+146378us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.616043 (+101055us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.712986 (+ 96943us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2421576}
I20251024 19:48:04.721599 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet 6ec5c7bf36874423b4790c6169da8bd0 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:04.721767 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 336) took 2733 ms. Trace:
I20251024 19:48:04.721869 1775 rpcz_store.cc:276] 1024 19:48:01.988011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:01.988191 (+ 180us) service_pool.cc:225] Handling call
1024 19:48:04.721524 (+2733333us) ts_tablet_manager.cc:1151] Deleting tablet 1edc55caab904722bf19a1c71df601cf
1024 19:48:04.721538 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.721702 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2733032}
I20251024 19:48:04.722281 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 337) took 2705 ms. Trace:
I20251024 19:48:04.722368 1775 rpcz_store.cc:276] 1024 19:48:02.016811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.016872 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:04.722120 (+2705248us) ts_tablet_manager.cc:1151] Deleting tablet e3b4e85f741849b2a6a3e8574396ff88
1024 19:48:04.722131 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.722221 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2704977}
I20251024 19:48:04.722669 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9b89a90bb4ca447fbaf34fe39001c261
I20251024 19:48:04.722714 1775 tablet_replica.cc:333] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:04.722965 1775 raft_consensus.cc:2243] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:04.724015 1775 raft_consensus.cc:2272] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:04.724047 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9aa983432fce4338b09f4182277c9825"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.724619 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9aa983432fce4338b09f4182277c9825 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.724823 1775 ts_tablet_manager.cc:1916] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:04.726521 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_928046849b864c76b36b8a2526f9bc5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.727008 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_928046849b864c76b36b8a2526f9bc5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.728478 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet f05ad092936b4bafa3ae795904c2df78 (table default.Table_99584fc98c7e4f0f853914cc3dda2b12 [id=0704c52aae2c421789a94e4391b9ed2a]) successfully deleted
W20251024 19:48:04.728715 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 1edc55caab904722bf19a1c71df601cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1edc55caab904722bf19a1c71df601cf already in progress: creating tablet
I20251024 19:48:04.728681 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b89a90bb4ca447fbaf34fe39001c261 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:04.728891 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet e3b4e85f741849b2a6a3e8574396ff88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3b4e85f741849b2a6a3e8574396ff88 already in progress: creating tablet
I20251024 19:48:04.729212 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be21ca442dee4204a8b15f5a2dd9530b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.729720 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b89a90bb4ca447fbaf34fe39001c261 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.729912 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cb7ec9294d4f438a042451f669b98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.730283 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b89a90bb4ca447fbaf34fe39001c261 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:04.730463 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1295c99836d14a1cb0002e6c6aaee1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.730952 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8e8dbca79f4d72a2de51f105b209e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.731410 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25ced70ff40d4149bb92e8b3767ec38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.731657 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81f8393f78204860bd854f4872099387" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:04.731817 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50d919a0b3d74e9a97d136a84d50e24b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:04.731885 1372 raft_consensus.cc:2468] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:04.731990 1371 raft_consensus.cc:3060] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.733822 1776 ts_tablet_manager.cc:1929] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:04.733969 1776 log.cc:1199] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/897db285f5c84930a5b438be1cb4eb96
I20251024 19:48:04.734475 1776 ts_tablet_manager.cc:1950] T 897db285f5c84930a5b438be1cb4eb96 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:04.737296 1447 leader_election.cc:304] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.737862 2224 raft_consensus.cc:2804] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:04.737984 2224 raft_consensus.cc:493] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:04.738065 2224 raft_consensus.cc:3060] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:04.739048 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_747b358db68e43159e1616089b57313a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.742509 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 25ced70ff40d4149bb92e8b3767ec38f (DEFAULT_TABLE table=default.Table_b9d913c84cec4d508db7fe4c3dc63e27 [id=eacc2d37d57f4d588101cb545db90c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.742938 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25ced70ff40d4149bb92e8b3767ec38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.747581 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 25ced70ff40d4149bb92e8b3767ec38f (DEFAULT_TABLE table=default.Table_b9d913c84cec4d508db7fe4c3dc63e27 [id=eacc2d37d57f4d588101cb545db90c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.750073 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 3da81c18d9264f38966944ad13f256d8 (DEFAULT_TABLE table=default.Table_4c1febbedc3448728852a72f7f5d2d44 [id=184c56c46f404346887cdb19db598f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.750442 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da81c18d9264f38966944ad13f256d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.751617 1650 raft_consensus.cc:3060] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.754091 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 94b35252e63f4a8ba305cc029ddb5ae8 (DEFAULT_TABLE table=default.Table_c13d19a355a24f2598c8d5741bbe8521 [id=f49bfd0c428d44f69a3d97a6c51a380a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.754550 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b35252e63f4a8ba305cc029ddb5ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.756246 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 3da81c18d9264f38966944ad13f256d8 (DEFAULT_TABLE table=default.Table_4c1febbedc3448728852a72f7f5d2d44 [id=184c56c46f404346887cdb19db598f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.756707 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da81c18d9264f38966944ad13f256d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.758143 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 94b35252e63f4a8ba305cc029ddb5ae8 (DEFAULT_TABLE table=default.Table_c13d19a355a24f2598c8d5741bbe8521 [id=f49bfd0c428d44f69a3d97a6c51a380a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.758674 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b35252e63f4a8ba305cc029ddb5ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.760432 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 3da81c18d9264f38966944ad13f256d8 (DEFAULT_TABLE table=default.Table_4c1febbedc3448728852a72f7f5d2d44 [id=184c56c46f404346887cdb19db598f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.761061 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 94b35252e63f4a8ba305cc029ddb5ae8 (DEFAULT_TABLE table=default.Table_c13d19a355a24f2598c8d5741bbe8521 [id=f49bfd0c428d44f69a3d97a6c51a380a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.763237 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 163ac050c8574615aef5061f563cfc0a (DEFAULT_TABLE table=default.Table_63c72c09261e44538e2cb5586550b43f [id=11a689b71c954aa4b8ac924fb28dbae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.763612 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163ac050c8574615aef5061f563cfc0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.767707 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 163ac050c8574615aef5061f563cfc0a (DEFAULT_TABLE table=default.Table_63c72c09261e44538e2cb5586550b43f [id=11a689b71c954aa4b8ac924fb28dbae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.767828 2246 raft_consensus.cc:515] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.768107 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163ac050c8574615aef5061f563cfc0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.768827 2246 leader_election.cc:290] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:04.769441 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c357a2a81064b3891afda6d252898b9" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:04.769735 1510 raft_consensus.cc:3060] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.770673 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c357a2a81064b3891afda6d252898b9" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:04.771059 1370 raft_consensus.cc:3060] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.774864 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3cf4596b333244cc9e9e35a128721e4a
I20251024 19:48:04.775583 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_adcb658ebf79468fb55f7635cf442d29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.776965 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adcb658ebf79468fb55f7635cf442d29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.777666 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf4596b333244cc9e9e35a128721e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.776233 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9ea4b3e0bf8d41ce844677ff6dc9ddfe" } new_table_name: "default.Table_93cc576cc317428c88108b05bf8efc6d" modify_external_catalogs: true
I20251024 19:48:04.781369 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf4596b333244cc9e9e35a128721e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:04.781510 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf4596b333244cc9e9e35a128721e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.782605 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 163ac050c8574615aef5061f563cfc0a (DEFAULT_TABLE table=default.Table_63c72c09261e44538e2cb5586550b43f [id=11a689b71c954aa4b8ac924fb28dbae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.782946 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50d919a0b3d74e9a97d136a84d50e24b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
W20251024 19:48:04.783540 1444 leader_election.cc:343] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.784854 1625 tablet_service.cc:1505] Processing CreateTablet for tablet b3e9960b7821421aa23e7ef5389f894a (DEFAULT_TABLE table=default.Table_b5a5283ea815484a91f644c13a1826b7 [id=28922f53a4d14fe2b8fdb29c01a5dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.785275 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3e9960b7821421aa23e7ef5389f894a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.786890 1350 tablet_service.cc:1505] Processing CreateTablet for tablet b3e9960b7821421aa23e7ef5389f894a (DEFAULT_TABLE table=default.Table_b5a5283ea815484a91f644c13a1826b7 [id=28922f53a4d14fe2b8fdb29c01a5dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.787495 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 74029a7d7cdc4503a833c1758887b684 (DEFAULT_TABLE table=default.Table_4aaaf3ef7e68447f8e9c6576c7405f5a [id=d35a25cbeefd47d39a5cba0bed321da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.790143 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 74029a7d7cdc4503a833c1758887b684 (DEFAULT_TABLE table=default.Table_4aaaf3ef7e68447f8e9c6576c7405f5a [id=d35a25cbeefd47d39a5cba0bed321da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.790575 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74029a7d7cdc4503a833c1758887b684. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.790362 1488 tablet_service.cc:1505] Processing CreateTablet for tablet b3e9960b7821421aa23e7ef5389f894a (DEFAULT_TABLE table=default.Table_b5a5283ea815484a91f644c13a1826b7 [id=28922f53a4d14fe2b8fdb29c01a5dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.791265 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3e9960b7821421aa23e7ef5389f894a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.791308 1483 tablet_service.cc:1505] Processing CreateTablet for tablet f88390f2534e4799a6eaa56aa4245e97 (DEFAULT_TABLE table=default.Table_edbcecddf69d4de8af2cf58ac7b12902 [id=7b26b4191a9d488abc502ae21bdf2ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.791817 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f88390f2534e4799a6eaa56aa4245e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.794082 1344 tablet_service.cc:1505] Processing CreateTablet for tablet f88390f2534e4799a6eaa56aa4245e97 (DEFAULT_TABLE table=default.Table_edbcecddf69d4de8af2cf58ac7b12902 [id=7b26b4191a9d488abc502ae21bdf2ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.801693 1618 tablet_service.cc:1505] Processing CreateTablet for tablet f88390f2534e4799a6eaa56aa4245e97 (DEFAULT_TABLE table=default.Table_edbcecddf69d4de8af2cf58ac7b12902 [id=7b26b4191a9d488abc502ae21bdf2ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.802251 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f88390f2534e4799a6eaa56aa4245e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.807317 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 74029a7d7cdc4503a833c1758887b684 (DEFAULT_TABLE table=default.Table_4aaaf3ef7e68447f8e9c6576c7405f5a [id=d35a25cbeefd47d39a5cba0bed321da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:04.808104 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74029a7d7cdc4503a833c1758887b684. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.813385 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_edbcecddf69d4de8af2cf58ac7b12902" } new_table_name: "default.Table_0e883c73a01f43fb99a5ad745839fae3" modify_external_catalogs: true
I20251024 19:48:04.814738 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ebcc09d8e029451d8fb6b6601ad16087
I20251024 19:48:04.815500 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d6de725f0921470e9db900df3d8a53dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:04.815737 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 404) took 1521 ms. Trace:
W20251024 19:48:04.815902 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6de725f0921470e9db900df3d8a53dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.815883 1776 rpcz_store.cc:276] 1024 19:48:03.294311 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.294562 (+ 251us) service_pool.cc:225] Handling call
1024 19:48:04.534760 (+1240198us) ts_tablet_manager.cc:1151] Deleting tablet 897db285f5c84930a5b438be1cb4eb96
1024 19:48:04.534769 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.629385 (+ 94616us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.733681 (+104296us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:04.815652 (+ 81971us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1239940}
I20251024 19:48:04.816417 1776 tablet_replica.cc:333] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:04.816686 1776 raft_consensus.cc:2243] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:04.816820 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet c04082bd1b444b468c177bef239701c6
I20251024 19:48:04.816937 1776 raft_consensus.cc:2272] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:04.817487 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 6ec5c7bf36874423b4790c6169da8bd0
I20251024 19:48:04.817816 1190 catalog_manager.cc:6719] ReplaceTablet: tablet 6ec5c7bf36874423b4790c6169da8bd0 deleted and replaced by tablet 7712392f200842c4b728e019d6a3cfb0
I20251024 19:48:04.818673 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 897db285f5c84930a5b438be1cb4eb96 (table default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]) successfully deleted
I20251024 19:48:04.819185 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet ebcc09d8e029451d8fb6b6601ad16087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.819593 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet c04082bd1b444b468c177bef239701c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.819988 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec5c7bf36874423b4790c6169da8bd0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7712392f200842c4b728e019d6a3cfb0) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.820603 1776 ts_tablet_manager.cc:1916] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:04.829334 1371 raft_consensus.cc:2468] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:04.833271 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet ebcc09d8e029451d8fb6b6601ad16087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.833592 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet c04082bd1b444b468c177bef239701c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.833837 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec5c7bf36874423b4790c6169da8bd0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7712392f200842c4b728e019d6a3cfb0) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.834704 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet ebcc09d8e029451d8fb6b6601ad16087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:04.834972 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet c04082bd1b444b468c177bef239701c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:04.835178 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec5c7bf36874423b4790c6169da8bd0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7712392f200842c4b728e019d6a3cfb0) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:04.835845 1447 leader_election.cc:304] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.836424 2173 raft_consensus.cc:2804] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:04.836627 2173 raft_consensus.cc:697] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:04.836985 2173 consensus_queue.cc:237] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.839232 2130 raft_consensus.cc:493] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.839443 2130 raft_consensus.cc:515] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.840289 2130 leader_election.cc:290] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:04.840744 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd52295b14b14e70bbdcb45f1bd849e4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:04.841763 1308 leader_election.cc:343] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:04.843617 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab488e4971874a16b2e052c84c250777 consensus state has no leader
I20251024 19:48:04.843940 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25ced70ff40d4149bb92e8b3767ec38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.844480 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f88390f2534e4799a6eaa56aa4245e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.844684 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3e9960b7821421aa23e7ef5389f894a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.849155 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74029a7d7cdc4503a833c1758887b684. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.852855 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b35252e63f4a8ba305cc029ddb5ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.854015 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2cbb0aad326e4bae85d4c5cb833150c6" } new_table_name: "default.Table_147e888401aa40e5a60d97884b36cb98" modify_external_catalogs: true
I20251024 19:48:04.856070 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_14c1f0af1332491a8ad3d0a6252f840d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.856762 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14c1f0af1332491a8ad3d0a6252f840d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.857762 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163ac050c8574615aef5061f563cfc0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.858273 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da81c18d9264f38966944ad13f256d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:04.862934 2141 raft_consensus.cc:515] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.864526 2141 leader_election.cc:290] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:04.866463 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913534ab88dd459e9cac2faebb5492a1" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
W20251024 19:48:04.867110 1444 leader_election.cc:343] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.875121 2141 consensus_queue.cc:1048] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:04.886073 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3bc26ab1c8fd48c682794066b1d61d4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.888062 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bc26ab1c8fd48c682794066b1d61d4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.890064 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_da1c2655e0de479e91cf0a21e1a728b5" } modify_external_catalogs: true
I20251024 19:48:04.890969 2246 consensus_queue.cc:1048] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:04.892431 2020 raft_consensus.cc:493] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:04.892585 2020 raft_consensus.cc:515] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.893257 2020 leader_election.cc:290] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:04.887775 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913534ab88dd459e9cac2faebb5492a1" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:04.894094 1371 raft_consensus.cc:3060] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:04.894268 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_da1c2655e0de479e91cf0a21e1a728b5" } modify_external_catalogs: true
W20251024 19:48:04.895875 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f88390f2534e4799a6eaa56aa4245e97 consensus state has no leader
I20251024 19:48:04.900084 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a7e76a3cac443ec8d2ab466075aae6b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:04.900390 1372 raft_consensus.cc:2468] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:04.901070 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_aa454e7717484495b5d4d77a0951c716" } new_table_name: "default.Table_c367e7fcf06a48fcbc2311c7a4998cac" modify_external_catalogs: true
I20251024 19:48:04.902009 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a7e76a3cac443ec8d2ab466075aae6b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:04.902645 1444 leader_election.cc:343] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.903424 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c13d19a355a24f2598c8d5741bbe8521" } new_table_name: "default.Table_a86db858d09749a9a4706c761c20ee9a" modify_external_catalogs: true
I20251024 19:48:04.904318 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd52295b14b14e70bbdcb45f1bd849e4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:04.904848 1447 leader_election.cc:304] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
W20251024 19:48:04.907521 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d04a9b54bd6446d8deb64292cef672d consensus state has no leader
I20251024 19:48:04.908599 2141 raft_consensus.cc:2804] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:04.908762 2141 raft_consensus.cc:493] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:04.908947 2141 raft_consensus.cc:3060] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:04.909157 1305 leader_election.cc:343] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:04.909294 1305 leader_election.cc:304] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:04.913336 2130 raft_consensus.cc:2749] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:04.930408 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_93cc576cc317428c88108b05bf8efc6d" } modify_external_catalogs: true
I20251024 19:48:04.930758 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_93cc576cc317428c88108b05bf8efc6d" } modify_external_catalogs: true
I20251024 19:48:04.934204 1767 tablet_bootstrap.cc:492] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:04.934497 1767 ts_tablet_manager.cc:1403] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.351s user 0.031s sys 0.078s
I20251024 19:48:04.935420 1767 raft_consensus.cc:359] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.935771 1767 raft_consensus.cc:385] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:04.935914 1767 raft_consensus.cc:740] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:04.936317 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 6424b2347ef642feb38a9596f2d2be1d
I20251024 19:48:04.936337 1767 consensus_queue.cc:260] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:04.937096 1767 ts_tablet_manager.cc:1434] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:04.937140 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 6424b2347ef642feb38a9596f2d2be1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.937506 1767 tablet_bootstrap.cc:492] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:04.938356 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 6424b2347ef642feb38a9596f2d2be1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.939117 2246 consensus_queue.cc:1048] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:04.939435 1767 tablet_bootstrap.cc:654] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:04.959477 1305 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 48860us
I20251024 19:48:04.960238 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 6424b2347ef642feb38a9596f2d2be1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:04.963076 1586 consensus_peers.cc:597] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9623d7baf0174a019e2353a5effef081. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:04.964818 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95ae017a2e084ccabfa69c9a5befe0e7 has no consensus state
W20251024 19:48:04.966109 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 94b35252e63f4a8ba305cc029ddb5ae8 consensus state has no leader
I20251024 19:48:04.977892 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet 3fd8aa1d9aca429dbfe46f472ebbf53b from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:04.979857 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ab488e4971874a16b2e052c84c250777
I20251024 19:48:04.982656 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet ab488e4971874a16b2e052c84c250777 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:04.985378 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_bc114bace73742c19c6d263e61d2bcdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:04.986094 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc114bace73742c19c6d263e61d2bcdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:04.986629 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet ab488e4971874a16b2e052c84c250777 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:04.988204 1768 tablet_bootstrap.cc:492] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:04.988255 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_aa7c283553a841e9ab2fa21fb6485a11" } modify_external_catalogs: true
I20251024 19:48:04.988540 1768 ts_tablet_manager.cc:1403] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.289s user 0.006s sys 0.001s
I20251024 19:48:04.988598 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_aa7c283553a841e9ab2fa21fb6485a11" } modify_external_catalogs: true
I20251024 19:48:04.990888 1370 raft_consensus.cc:2468] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:04.994702 1586 leader_election.cc:304] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:04.995043 2146 raft_consensus.cc:2804] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:04.995196 2146 raft_consensus.cc:697] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:04.995527 2146 consensus_queue.cc:237] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.008916 2146 consensus_queue.cc:1048] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:05.019098 2146 raft_consensus.cc:493] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.019430 2146 raft_consensus.cc:515] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.020956 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "346383e85cc64b3a8da70a71868ceccc" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:05.021250 1508 raft_consensus.cc:2468] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:05.022156 1586 leader_election.cc:304] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:05.022586 2246 raft_consensus.cc:2804] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:05.022686 2246 raft_consensus.cc:493] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:05.022801 2246 raft_consensus.cc:3060] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.026383 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3b9aa7dd64334765afe410d802b10292" } modify_external_catalogs: true
I20251024 19:48:05.026607 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3b9aa7dd64334765afe410d802b10292" } modify_external_catalogs: true
I20251024 19:48:05.026753 2146 leader_election.cc:290] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.027243 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d1c0f808bdec4b94825c72f0043421cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.027381 1586 leader_election.cc:343] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 346383e85cc64b3a8da70a71868ceccc
W20251024 19:48:05.027829 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1c0f808bdec4b94825c72f0043421cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.029285 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fcba3502a1824559a30dc2ffa6a9d6b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.029873 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcba3502a1824559a30dc2ffa6a9d6b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.034169 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f052d9f6feec4b35be9654644bbb59c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.034962 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f052d9f6feec4b35be9654644bbb59c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.036504 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_890d1e70131f42a1a8b2c4870dd03e7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.037220 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_890d1e70131f42a1a8b2c4870dd03e7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.038034 1371 raft_consensus.cc:2468] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:05.044658 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet b5473699c1b647f8b86ee43fdb56bdfd from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:05.046048 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ae2a66ac44f249a2b5335fa64ff1c99c" } modify_external_catalogs: true
I20251024 19:48:05.046274 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ae2a66ac44f249a2b5335fa64ff1c99c" } modify_external_catalogs: true
I20251024 19:48:05.046725 1767 tablet_bootstrap.cc:492] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.046904 1767 ts_tablet_manager.cc:1403] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.109s user 0.010s sys 0.012s
I20251024 19:48:05.054939 1767 raft_consensus.cc:359] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.058403 1767 raft_consensus.cc:385] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.058717 1767 raft_consensus.cc:740] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.059273 1767 consensus_queue.cc:260] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.061841 2246 raft_consensus.cc:515] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.065500 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_564c9b80558d45a084b6a36dc542f1d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.066231 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_564c9b80558d45a084b6a36dc542f1d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.066243 2246 leader_election.cc:290] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.067400 1767 ts_tablet_manager.cc:1434] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251024 19:48:05.067658 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5fd79a379e3b43aeb80de8af15bf4427"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.068159 1586 leader_election.cc:343] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 346383e85cc64b3a8da70a71868ceccc
I20251024 19:48:05.068260 1767 tablet_bootstrap.cc:492] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:48:05.068377 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fd79a379e3b43aeb80de8af15bf4427 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.069954 1767 tablet_bootstrap.cc:654] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.074574 1333 tablet_service.cc:1552] Processing DeleteTablet for tablet ab488e4971874a16b2e052c84c250777 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:05.083247 1767 tablet_bootstrap.cc:492] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.083411 1767 ts_tablet_manager.cc:1403] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20251024 19:48:05.084290 1767 raft_consensus.cc:359] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.084474 1767 raft_consensus.cc:385] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.084564 1767 raft_consensus.cc:740] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.089738 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet a48249f1824a40b087b80a563816d485 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.093871 1767 consensus_queue.cc:260] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.094105 2246 raft_consensus.cc:493] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.094246 2246 raft_consensus.cc:515] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.094893 2246 leader_election.cc:290] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.095676 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d1933bb728845abad5e6b09b24868f0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:05.096197 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d6de725f0921470e9db900df3d8a53dd" } new_table_name: "default.Table_9db467b0e3b745678279c3f288a7c988" modify_external_catalogs: true
I20251024 19:48:05.101341 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5f6d82174b9d4f64bbf61f72e5b58a25" } new_table_name: "default.Table_715723e6de6a4ad0861818fa05311861" modify_external_catalogs: true
I20251024 19:48:05.101895 1447 leader_election.cc:304] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:05.102448 1767 ts_tablet_manager.cc:1434] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20251024 19:48:05.102718 2020 raft_consensus.cc:2804] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:05.102980 2020 raft_consensus.cc:697] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:05.103209 1767 tablet_bootstrap.cc:492] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.103284 2020 consensus_queue.cc:237] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.104265 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet b5473699c1b647f8b86ee43fdb56bdfd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.104867 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d1933bb728845abad5e6b09b24868f0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:05.105171 1511 raft_consensus.cc:2393] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 43bec2f03fc849099e254f6335e9d532 in this term.
I20251024 19:48:05.105252 1767 tablet_bootstrap.cc:654] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.105702 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "346383e85cc64b3a8da70a71868ceccc" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:05.105930 1508 raft_consensus.cc:3060] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.108212 2173 consensus_queue.cc:1048] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:05.110087 1586 leader_election.cc:304] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
W20251024 19:48:05.110276 1305 consensus_peers.cc:597] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f05ad092936b4bafa3ae795904c2df78. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:05.110638 2246 raft_consensus.cc:3060] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:05.113916 1308 consensus_peers.cc:597] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f05ad092936b4bafa3ae795904c2df78. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:05.117012 1767 tablet_bootstrap.cc:492] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.117182 1767 ts_tablet_manager.cc:1403] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20251024 19:48:05.118079 1767 raft_consensus.cc:359] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.118288 1767 raft_consensus.cc:385] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.118371 1767 raft_consensus.cc:740] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.121182 1774 ts_tablet_manager.cc:1929] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:05.121414 1774 log.cc:1199] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/b9b1e0338fd14634a1e921735ae61289
I20251024 19:48:05.121932 1774 ts_tablet_manager.cc:1950] T b9b1e0338fd14634a1e921735ae61289 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:05.122352 1768 raft_consensus.cc:359] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.122633 1768 raft_consensus.cc:385] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.122740 1768 raft_consensus.cc:740] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.123190 1768 consensus_queue.cc:260] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:05.123737 1447 consensus_peers.cc:597] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e20881e400cf4fac914b70afc79c084f. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:05.125432 1767 consensus_queue.cc:260] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.126749 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_16e10ff29f654c72802003f0c316b89f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.128022 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16e10ff29f654c72802003f0c316b89f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.129698 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d1933bb728845abad5e6b09b24868f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.133563 1767 ts_tablet_manager.cc:1434] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251024 19:48:05.133916 1767 tablet_bootstrap.cc:492] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.135573 1767 tablet_bootstrap.cc:654] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.137079 1768 ts_tablet_manager.cc:1434] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.148s user 0.003s sys 0.000s
I20251024 19:48:05.137616 1768 tablet_bootstrap.cc:492] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:05.140048 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_939063dc1bea4e6d9d22b04cafce746b" } new_table_name: "default.Table_74e4f24f12db403686548924c83c5191" modify_external_catalogs: true
I20251024 19:48:05.140710 1768 tablet_bootstrap.cc:654] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.145042 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 307) took 2994 ms. Trace:
I20251024 19:48:05.145161 1774 rpcz_store.cc:276] 1024 19:48:02.151052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.151104 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:04.630992 (+2479888us) ts_tablet_manager.cc:1151] Deleting tablet b9b1e0338fd14634a1e921735ae61289
1024 19:48:04.631003 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.925617 (+294614us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.121022 (+195405us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.145004 (+ 23982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2479620}
I20251024 19:48:05.145563 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet b9b1e0338fd14634a1e921735ae61289 (table default.Table_f02338f14f2b44b3b89757b7edc59e32 [id=3e4514469cd64a8b86d9e44608c186ba]) successfully deleted
I20251024 19:48:05.150007 1774 tablet_replica.cc:333] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:05.150224 1774 raft_consensus.cc:2243] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:05.150393 1774 raft_consensus.cc:2272] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:05.151136 1776 ts_tablet_manager.cc:1929] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:05.151324 1776 log.cc:1199] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/31647a48fb834c80b6920c756a94250d
I20251024 19:48:05.151710 2246 raft_consensus.cc:2749] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:05.151952 1766 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.033s sys 0.110s
W20251024 19:48:05.152180 1766 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.558s user 0.033s sys 0.110s
I20251024 19:48:05.153409 1510 raft_consensus.cc:2468] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:05.155031 1767 tablet_bootstrap.cc:492] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.155166 1767 ts_tablet_manager.cc:1403] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251024 19:48:05.156395 1767 raft_consensus.cc:359] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.156638 1767 raft_consensus.cc:385] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.156845 1767 raft_consensus.cc:740] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.157192 1767 consensus_queue.cc:260] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.157974 1767 ts_tablet_manager.cc:1434] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251024 19:48:05.158282 1767 tablet_bootstrap.cc:492] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.160008 1767 tablet_bootstrap.cc:654] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.163800 1774 ts_tablet_manager.cc:1916] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:05.167768 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98dd8bacd8374f469799872859db4206 has no consensus state
I20251024 19:48:05.169600 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_94d237360e434747ba2dcdf09d4367cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:05.170307 1775 ts_tablet_manager.cc:1929] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251024 19:48:05.170372 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10891ff5bf0d4b93bcab9faa5f7346f1 consensus state has no leader
I20251024 19:48:05.170442 1775 log.cc:1199] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/c292ce055def4964ab143252c6b2b87e
I20251024 19:48:05.170562 1766 tablet_bootstrap.cc:492] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:05.170734 1766 ts_tablet_manager.cc:1403] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.580s user 0.034s sys 0.113s
I20251024 19:48:05.170696 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_86b9f6987402431196ad8064631c34bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:05.170928 1775 ts_tablet_manager.cc:1950] T c292ce055def4964ab143252c6b2b87e P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:05.171518 1766 raft_consensus.cc:359] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.171715 1766 raft_consensus.cc:385] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.171779 1766 raft_consensus.cc:740] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.172001 1766 consensus_queue.cc:260] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.172523 1766 ts_tablet_manager.cc:1434] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:05.172691 1776 ts_tablet_manager.cc:1950] T 31647a48fb834c80b6920c756a94250d P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:05.172881 1766 tablet_bootstrap.cc:492] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:05.173192 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_61b7f6c428aa4bd9a65c1d66171c59c9" } modify_external_catalogs: true
I20251024 19:48:05.173367 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_61b7f6c428aa4bd9a65c1d66171c59c9" } modify_external_catalogs: true
I20251024 19:48:05.174558 1766 tablet_bootstrap.cc:654] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.177944 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 405) took 1883 ms. Trace:
I20251024 19:48:05.178098 1776 rpcz_store.cc:276] 1024 19:48:03.294513 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.294859 (+ 346us) service_pool.cc:225] Handling call
1024 19:48:04.816289 (+1521430us) ts_tablet_manager.cc:1151] Deleting tablet 31647a48fb834c80b6920c756a94250d
1024 19:48:04.816302 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:04.989862 (+173560us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.150984 (+161122us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.177894 (+ 26910us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1521269}
I20251024 19:48:05.178566 1776 tablet_replica.cc:333] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:05.178851 1776 raft_consensus.cc:2243] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:05.178907 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 31647a48fb834c80b6920c756a94250d (table default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]) successfully deleted
I20251024 19:48:05.179162 1776 raft_consensus.cc:2272] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:05.184559 1766 tablet_bootstrap.cc:492] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:05.184734 1766 ts_tablet_manager.cc:1403] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20251024 19:48:05.185549 1766 raft_consensus.cc:359] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.185786 1766 raft_consensus.cc:385] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.185878 1766 raft_consensus.cc:740] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.185915 2272 raft_consensus.cc:493] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.186110 1766 consensus_queue.cc:260] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.186062 2272 raft_consensus.cc:515] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.186622 1766 ts_tablet_manager.cc:1434] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251024 19:48:05.186882 2272 leader_election.cc:290] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:05.186939 1766 tablet_bootstrap.cc:492] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:05.187485 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db17d9aff6542b6b94c09f872a4cceb" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:05.188638 1766 tablet_bootstrap.cc:654] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:05.188769 1308 leader_election.cc:343] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:05.189736 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db17d9aff6542b6b94c09f872a4cceb" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:05.190532 1305 leader_election.cc:343] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:05.190675 1767 tablet_bootstrap.cc:492] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.190647 1305 leader_election.cc:304] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:05.190850 1767 ts_tablet_manager.cc:1403] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.006s
I20251024 19:48:05.191253 2272 raft_consensus.cc:2749] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:05.191591 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7abb98af07f64d6899a9c6d7b2d84aa2" } new_table_name: "default.Table_aec0055f05e645e5a429d88125c4fb6f" modify_external_catalogs: true
I20251024 19:48:05.191661 1767 raft_consensus.cc:359] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.191789 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 50d919a0b3d74e9a97d136a84d50e24b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.191881 1767 raft_consensus.cc:385] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.191947 1767 raft_consensus.cc:740] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.192201 1767 consensus_queue.cc:260] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.192834 1767 ts_tablet_manager.cc:1434] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:05.193264 1767 tablet_bootstrap.cc:492] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.195016 1767 tablet_bootstrap.cc:654] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.196223 1650 raft_consensus.cc:3060] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:05.197278 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86b9f6987402431196ad8064631c34bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:05.198319 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94d237360e434747ba2dcdf09d4367cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.202351 1766 tablet_bootstrap.cc:492] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:05.202538 1766 ts_tablet_manager.cc:1403] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.002s
I20251024 19:48:05.203414 1766 raft_consensus.cc:359] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.203617 1766 raft_consensus.cc:385] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.203707 1766 raft_consensus.cc:740] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.203946 1766 consensus_queue.cc:260] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.204218 1767 tablet_bootstrap.cc:492] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.204396 1767 ts_tablet_manager.cc:1403] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251024 19:48:05.204456 1766 ts_tablet_manager.cc:1434] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:05.204778 1766 tablet_bootstrap.cc:492] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:05.205273 1767 raft_consensus.cc:359] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.205453 1767 raft_consensus.cc:385] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.206485 1766 tablet_bootstrap.cc:654] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:05.211491 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 505e4fb0755a4caaa462e5f5154bfee2 consensus state has no leader
I20251024 19:48:05.205556 1767 raft_consensus.cc:740] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.212154 1776 ts_tablet_manager.cc:1916] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:05.213893 1767 consensus_queue.cc:260] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.214336 1766 tablet_bootstrap.cc:492] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:05.214460 1767 ts_tablet_manager.cc:1434] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20251024 19:48:05.214488 1766 ts_tablet_manager.cc:1403] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20251024 19:48:05.215250 1766 raft_consensus.cc:359] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.215425 1766 raft_consensus.cc:385] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.215642 1766 raft_consensus.cc:740] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.215941 1766 consensus_queue.cc:260] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.216507 1766 ts_tablet_manager.cc:1434] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:05.216859 1766 tablet_bootstrap.cc:492] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:05.220863 1766 tablet_bootstrap.cc:654] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.221943 1767 tablet_bootstrap.cc:492] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.223445 1766 tablet_bootstrap.cc:492] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:05.223644 1766 ts_tablet_manager.cc:1403] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20251024 19:48:05.224148 1767 tablet_bootstrap.cc:654] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.224479 1766 raft_consensus.cc:359] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.224802 1766 raft_consensus.cc:385] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.224946 1766 raft_consensus.cc:740] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
W20251024 19:48:05.225193 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eee675874516489f9e8dac41f70ac4e4 has no consensus state
I20251024 19:48:05.225325 1766 consensus_queue.cc:260] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.226133 1766 ts_tablet_manager.cc:1434] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251024 19:48:05.226593 1766 tablet_bootstrap.cc:492] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:05.227406 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_32d4a5b034974da0a745844ffcab9498" } new_table_name: "default.Table_e63d3e693e1148a7b8d2b5731b1027bc" modify_external_catalogs: true
I20251024 19:48:05.229169 1766 tablet_bootstrap.cc:654] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.231312 1767 tablet_bootstrap.cc:492] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.231523 1767 ts_tablet_manager.cc:1403] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20251024 19:48:05.232581 1767 raft_consensus.cc:359] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.234396 2173 consensus_queue.cc:1048] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:05.235859 2272 raft_consensus.cc:493] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.236111 2272 raft_consensus.cc:515] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.236580 1766 tablet_bootstrap.cc:492] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:05.236833 1766 ts_tablet_manager.cc:1403] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20251024 19:48:05.237075 2272 leader_election.cc:290] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:05.237749 2224 raft_consensus.cc:515] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.238206 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a1f1892bd3b4776adc7c2519fcdf411" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:05.238490 2224 leader_election.cc:290] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.238449 1766 raft_consensus.cc:359] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.238662 1766 raft_consensus.cc:385] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.238777 1766 raft_consensus.cc:740] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.239058 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a1f1892bd3b4776adc7c2519fcdf411" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:05.239041 1766 consensus_queue.cc:260] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.239698 1766 ts_tablet_manager.cc:1434] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251024 19:48:05.239778 1305 leader_election.cc:343] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:05.240048 1766 tablet_bootstrap.cc:492] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:05.241935 1766 tablet_bootstrap.cc:654] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.232978 1767 raft_consensus.cc:385] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.243252 1767 raft_consensus.cc:740] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.244191 1767 consensus_queue.cc:260] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.246238 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7dc53b918ca7443b807133df217c0f75" } modify_external_catalogs: true
I20251024 19:48:05.246443 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7dc53b918ca7443b807133df217c0f75" } modify_external_catalogs: true
I20251024 19:48:05.246515 1767 ts_tablet_manager.cc:1434] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.015s user 0.001s sys 0.002s
I20251024 19:48:05.246869 1767 tablet_bootstrap.cc:492] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.247673 1768 tablet_bootstrap.cc:492] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:05.247814 1768 ts_tablet_manager.cc:1403] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.002s
I20251024 19:48:05.238504 1510 raft_consensus.cc:2468] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:05.253949 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81f8393f78204860bd854f4872099387" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:05.254163 1371 raft_consensus.cc:3060] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.255283 1308 leader_election.cc:304] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:05.257483 2272 raft_consensus.cc:2804] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:05.257737 2272 raft_consensus.cc:493] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:05.257906 2272 raft_consensus.cc:3060] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.260352 1766 tablet_bootstrap.cc:492] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:05.260532 1766 ts_tablet_manager.cc:1403] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.010s
I20251024 19:48:05.261672 2020 consensus_queue.cc:1048] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:05.270116 1766 raft_consensus.cc:359] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.270643 1766 raft_consensus.cc:385] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.270758 1766 raft_consensus.cc:740] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.270998 1766 consensus_queue.cc:260] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.271593 1766 ts_tablet_manager.cc:1434] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251024 19:48:05.277050 1766 tablet_bootstrap.cc:492] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:05.278920 1766 tablet_bootstrap.cc:654] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.286495 1192 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]). Expected version 1 got 0
I20251024 19:48:05.287390 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 505e4fb0755a4caaa462e5f5154bfee2 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:05.291839 1185 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]). Expected version 1 got 0
W20251024 19:48:05.293907 2013 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:52586 (request call id 0) took 887 ms (client timeout 1000 ms). Trace:
W20251024 19:48:05.294071 2013 rpcz_store.cc:269] 1024 19:48:04.406090 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.406175 (+ 85us) service_pool.cc:225] Handling call
1024 19:48:05.293839 (+887664us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251024 19:48:05.295686 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 1d04a9b54bd6446d8deb64292cef672d (table default.Table_147e888401aa40e5a60d97884b36cb98 [id=1acdc5d0bfb94e0faf47055b20d803b0]). Expected version 1 got 0
I20251024 19:48:05.297420 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 8876b7b38e744212a3722dbe5bcc5596 (DEFAULT_TABLE table=default.Table_7ab38232d57d40fea8917ef3a67df1ce [id=999ae3ac62d84ef586c8b43a6cdb9bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.298552 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8876b7b38e744212a3722dbe5bcc5596. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.302608 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet dc1b6f0a377945089c50012f74d38564 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ca8f818440f4ae7a63c0901fcdcdf59) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:05.302942 2268 consensus_queue.cc:1048] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:05.306294 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 8876b7b38e744212a3722dbe5bcc5596 (DEFAULT_TABLE table=default.Table_7ab38232d57d40fea8917ef3a67df1ce [id=999ae3ac62d84ef586c8b43a6cdb9bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.306881 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8876b7b38e744212a3722dbe5bcc5596. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.255905 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81f8393f78204860bd854f4872099387" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:05.299054 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 8876b7b38e744212a3722dbe5bcc5596 (DEFAULT_TABLE table=default.Table_7ab38232d57d40fea8917ef3a67df1ce [id=999ae3ac62d84ef586c8b43a6cdb9bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.312677 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8876b7b38e744212a3722dbe5bcc5596. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:05.314284 1444 leader_election.cc:343] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:05.315364 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 593f5600e4274fce97e7bd33f025569f (DEFAULT_TABLE table=default.Table_998040d863404075b6422120ab4ce290 [id=b576f9ef53f54b8ea2cf3f520a5d7076]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.315855 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593f5600e4274fce97e7bd33f025569f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.316305 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 593f5600e4274fce97e7bd33f025569f (DEFAULT_TABLE table=default.Table_998040d863404075b6422120ab4ce290 [id=b576f9ef53f54b8ea2cf3f520a5d7076]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.316493 1766 tablet_bootstrap.cc:492] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:05.316645 1766 ts_tablet_manager.cc:1403] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.003s
I20251024 19:48:05.316704 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593f5600e4274fce97e7bd33f025569f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.317655 1766 raft_consensus.cc:359] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.317864 1766 raft_consensus.cc:385] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.317955 1766 raft_consensus.cc:740] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.318271 1766 consensus_queue.cc:260] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.318811 1766 ts_tablet_manager.cc:1434] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:05.319110 1766 tablet_bootstrap.cc:492] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:05.321157 1766 tablet_bootstrap.cc:654] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.322772 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 593f5600e4274fce97e7bd33f025569f (DEFAULT_TABLE table=default.Table_998040d863404075b6422120ab4ce290 [id=b576f9ef53f54b8ea2cf3f520a5d7076]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.323253 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593f5600e4274fce97e7bd33f025569f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.326083 1768 raft_consensus.cc:359] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.326337 1768 raft_consensus.cc:385] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.326470 1768 raft_consensus.cc:740] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.326638 1618 tablet_service.cc:1505] Processing CreateTablet for tablet a0a6a1cd6e314c20b55d857c1564d188 (DEFAULT_TABLE table=default.Table_415e7ca23c72439a9bebab742ea81150 [id=91564a7da2b147a2a682b31b67cf7ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.326895 1768 consensus_queue.cc:260] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.327073 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0a6a1cd6e314c20b55d857c1564d188. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.329829 1648 raft_consensus.cc:3060] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.332334 1768 ts_tablet_manager.cc:1434] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.084s user 0.002s sys 0.000s
I20251024 19:48:05.333060 1371 raft_consensus.cc:2468] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:05.333439 1766 tablet_bootstrap.cc:492] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:05.333652 1766 ts_tablet_manager.cc:1403] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20251024 19:48:05.334410 1766 raft_consensus.cc:359] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.334596 1766 raft_consensus.cc:385] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.334654 1766 raft_consensus.cc:740] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.334899 1766 consensus_queue.cc:260] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.335364 1766 ts_tablet_manager.cc:1434] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:05.335803 1766 tablet_bootstrap.cc:492] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:05.337828 1766 tablet_bootstrap.cc:654] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.339038 1447 leader_election.cc:304] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:05.339740 2224 raft_consensus.cc:2804] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:05.339911 2224 raft_consensus.cc:697] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:05.340090 2217 consensus_queue.cc:1048] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:05.340202 2224 consensus_queue.cc:237] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.342029 2141 raft_consensus.cc:515] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.342731 2141 leader_election.cc:290] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.343041 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a7e76a3cac443ec8d2ab466075aae6b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:05.343254 1650 raft_consensus.cc:3060] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.353197 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a7e76a3cac443ec8d2ab466075aae6b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:05.353462 1371 raft_consensus.cc:3060] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:05.361677 1486 consensus_meta.cc:287] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e: Time spent flushing consensus metadata: real 0.600s user 0.004s sys 0.000s
I20251024 19:48:05.362507 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 338) took 3327 ms. Trace:
I20251024 19:48:05.362886 1767 tablet_bootstrap.cc:654] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.363498 1775 rpcz_store.cc:276] 1024 19:48:02.034612 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.038050 (+ 3438us) service_pool.cc:225] Handling call
1024 19:48:04.722625 (+2684575us) ts_tablet_manager.cc:1151] Deleting tablet c292ce055def4964ab143252c6b2b87e
1024 19:48:04.722635 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.020298 (+297663us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.170140 (+149842us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.362467 (+192327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2684331}
I20251024 19:48:05.364058 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 339) took 3301 ms. Trace:
I20251024 19:48:05.365139 1775 rpcz_store.cc:276] 1024 19:48:02.062432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.062543 (+ 111us) service_pool.cc:225] Handling call
1024 19:48:05.363865 (+3301322us) ts_tablet_manager.cc:1151] Deleting tablet 3e6db58fac5e4af9b27bde6c223aa845
1024 19:48:05.363877 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.364023 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3301038}
I20251024 19:48:05.365480 1775 tablet_replica.cc:333] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
W20251024 19:48:05.365615 1474 consensus_meta.cc:287] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e: Time spent flushing consensus metadata: real 0.583s user 0.003s sys 0.000s
W20251024 19:48:05.366806 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3e6db58fac5e4af9b27bde6c223aa845 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e6db58fac5e4af9b27bde6c223aa845 already in progress: creating tablet
I20251024 19:48:05.367728 2141 raft_consensus.cc:493] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.369009 1775 raft_consensus.cc:2243] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:05.369557 1775 raft_consensus.cc:2272] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:05.345537 1488 consensus_meta.cc:287] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e: Time spent flushing consensus metadata: real 0.544s user 0.002s sys 0.001s
I20251024 19:48:05.370194 2141 raft_consensus.cc:515] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.371266 2141 leader_election.cc:290] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.372943 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76d61c3464b41519d7657796bd853ff" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:05.375223 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76d61c3464b41519d7657796bd853ff" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:05.377936 1767 tablet_bootstrap.cc:492] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.378440 1767 ts_tablet_manager.cc:1403] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.132s user 0.004s sys 0.000s
W20251024 19:48:05.379012 1444 leader_election.cc:343] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:05.379266 1767 raft_consensus.cc:359] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.379451 1767 raft_consensus.cc:385] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251024 19:48:05.379321 1447 leader_election.cc:343] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:05.379544 1767 raft_consensus.cc:740] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.379791 1767 consensus_queue.cc:260] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.380362 1767 ts_tablet_manager.cc:1434] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:05.380537 1447 leader_election.cc:304] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:05.380633 1767 tablet_bootstrap.cc:492] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.382544 1767 tablet_bootstrap.cc:654] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.385978 1775 ts_tablet_manager.cc:1916] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:05.389900 2141 raft_consensus.cc:2749] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:05.396034 1767 tablet_bootstrap.cc:492] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.396373 1767 ts_tablet_manager.cc:1403] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20251024 19:48:05.397167 1767 raft_consensus.cc:359] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.397397 1767 raft_consensus.cc:385] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.397485 1767 raft_consensus.cc:740] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.397774 1767 consensus_queue.cc:260] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.398434 1767 ts_tablet_manager.cc:1434] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:05.398922 1767 tablet_bootstrap.cc:492] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.400751 1767 tablet_bootstrap.cc:654] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.408213 2224 raft_consensus.cc:493] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.408411 2224 raft_consensus.cc:515] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.408882 1767 tablet_bootstrap.cc:492] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.409022 1767 ts_tablet_manager.cc:1403] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20251024 19:48:05.409178 2224 leader_election.cc:290] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.410184 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "037c8cc1e53447109fafc047c02f196f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:05.410782 1444 leader_election.cc:343] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:05.411094 1767 raft_consensus.cc:359] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.411404 1767 raft_consensus.cc:385] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.411465 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "037c8cc1e53447109fafc047c02f196f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:05.411547 1767 raft_consensus.cc:740] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.411823 1767 consensus_queue.cc:260] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.412490 1767 ts_tablet_manager.cc:1434] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:05.412779 1767 tablet_bootstrap.cc:492] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.414623 1767 tablet_bootstrap.cc:654] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:05.415922 1447 leader_election.cc:343] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:05.416147 1447 leader_election.cc:304] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:05.416499 2224 raft_consensus.cc:2749] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:05.422839 1767 tablet_bootstrap.cc:492] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.423017 1767 ts_tablet_manager.cc:1403] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20251024 19:48:05.423817 1767 raft_consensus.cc:359] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.424023 1767 raft_consensus.cc:385] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.424098 1767 raft_consensus.cc:740] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.424391 1767 consensus_queue.cc:260] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.424953 1767 ts_tablet_manager.cc:1434] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251024 19:48:05.425072 1187 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]). Expected version 1 got 0
I20251024 19:48:05.425248 1767 tablet_bootstrap.cc:492] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.426967 1767 tablet_bootstrap.cc:654] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.434420 1650 raft_consensus.cc:2468] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:05.434439 1371 raft_consensus.cc:2468] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:05.434934 1447 leader_election.cc:304] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:05.434898 2272 raft_consensus.cc:515] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.435329 1508 raft_consensus.cc:2468] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:05.435602 2272 leader_election.cc:290] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:05.436437 1934 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.252.195:44055 (request call id 47) took 1606 ms (client timeout 1000 ms). Trace:
W20251024 19:48:05.436563 1934 rpcz_store.cc:269] 1024 19:48:03.829808 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.829876 (+ 68us) service_pool.cc:225] Handling call
1024 19:48:05.435673 (+1605797us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x616000060a54. stack: 00007f81a26cd9f6 00007f81a26cd7b0 00007f81a16a93ad 0000000000983f04 0000000000983ecd 00007f81b19cb6bd 00007f81b19cb7b3 00007f81ae596441 00007f81b624c9dd 00007f81b63d3aa8 00007f81aceecca6 00007f81aceec9d5 00007f81a966927e 00007f81a966815e 00007f81a966e395 00007f81a966ecbf
1024 19:48:05.436328 (+ 655us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3691609984}
W20251024 19:48:05.439648 1933 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.252.194:54175 (request call id 49) took 1603 ms (client timeout 1000 ms). Trace:
I20251024 19:48:05.439723 2236 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc8f71a5b70749a4a3e3a599bcb3d22c)
W20251024 19:48:05.439778 1933 rpcz_store.cc:269] 1024 19:48:03.836000 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.836052 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:05.438591 (+1602539us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x616000060a54. stack: 00007f81a26cd9f6 00007f81a26cd7b0 00007f81a16a93ad 0000000000983f04 0000000000983ecd 00007f81b19cb6bd 00007f81b19cb7b3 00007f81ae596441 00007f81b624c9dd 00007f81b63d3aa8 00007f81aceecca6 00007f81aceec9d5 00007f81a966927e 00007f81a966815e 00007f81a966e395 00007f81a966ecbf
1024 19:48:05.439586 (+ 995us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3684084992}
I20251024 19:48:05.439927 2236 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:48:05.441139 1960 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52566 (request call id 3) took 1636 ms. Trace:
I20251024 19:48:05.445624 1586 leader_election.cc:304] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:05.446527 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a1f1892bd3b4776adc7c2519fcdf411" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:05.446584 1129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef8906492b1a4368936fac95c11e1f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 515 } } ignore_live_leader: false dest_uuid: "82e82847f8904b59b03f446b7fce6233" is_pre_election: true
I20251024 19:48:05.446728 1508 raft_consensus.cc:3060] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.449997 2224 raft_consensus.cc:2804] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:05.450222 2224 raft_consensus.cc:697] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:05.450562 2224 consensus_queue.cc:237] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.451297 2020 raft_consensus.cc:493] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.451450 2246 raft_consensus.cc:2804] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:05.451493 2020 raft_consensus.cc:515] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.451618 2246 raft_consensus.cc:697] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:05.452292 2020 leader_election.cc:290] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.452719 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "030e690ce4df4818a6450ae293b0ffbd" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:05.452935 1371 raft_consensus.cc:2468] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:05.452917 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a1f1892bd3b4776adc7c2519fcdf411" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:05.453090 1650 raft_consensus.cc:3060] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.453511 1447 leader_election.cc:304] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:05.452550 2246 consensus_queue.cc:237] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:05.453624 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a7e76a3cac443ec8d2ab466075aae6b consensus state has no leader
I20251024 19:48:05.456081 2272 consensus_queue.cc:1048] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:05.457808 2141 raft_consensus.cc:2804] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:05.457981 2141 raft_consensus.cc:493] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:05.458113 2141 raft_consensus.cc:3060] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.458280 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "030e690ce4df4818a6450ae293b0ffbd" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:05.458595 1648 raft_consensus.cc:2468] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:05.460443 1202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef8906492b1a4368936fac95c11e1f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 515 } } ignore_live_leader: false dest_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" is_pre_election: true
I20251024 19:48:05.460760 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 50d919a0b3d74e9a97d136a84d50e24b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:05.463807 2236 leader_election.cc:290] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82e82847f8904b59b03f446b7fce6233 (127.0.252.253:41375), dc8f71a5b70749a4a3e3a599bcb3d22c (127.0.252.252:36627)
I20251024 19:48:05.466467 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet f346a90a9efe442f82750336e3893deb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.468932 1916 leader_election.cc:304] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef8906492b1a4368936fac95c11e1f73; no voters: 82e82847f8904b59b03f446b7fce6233, dc8f71a5b70749a4a3e3a599bcb3d22c
I20251024 19:48:05.473062 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_dbcb9a24f3f34a91b79bbc952d4f0012"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.473666 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbcb9a24f3f34a91b79bbc952d4f0012 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.474785 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_783450f8b33e4b2b936136effef29fc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.475198 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_783450f8b33e4b2b936136effef29fc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.480167 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_86c533ff0d26405bb7a1c203c39d18a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.480592 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86c533ff0d26405bb7a1c203c39d18a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.482710 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a1f1892bd3b4776adc7c2519fcdf411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:05.491410 2236 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251024 19:48:05.491397 1935 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.252.193:56151 (request call id 48) took 1665 ms (client timeout 1000 ms). Trace:
W20251024 19:48:05.491803 1935 rpcz_store.cc:269] 1024 19:48:03.812742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.812796 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:05.469475 (+1656679us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x616000060a54. stack: 00007f81a26cd9f6 00007f81a26cd7b0 00007f81a16a93ad 0000000000983f04 0000000000983ecd 00007f81b19cb6bd 00007f81b19cb7b3 00007f81ae596441 00007f81b624c9dd 00007f81b63d3aa8 00007f81aceecca6 00007f81aceec9d5 00007f81a966927e 00007f81a966815e 00007f81a966e395 00007f81a966ecbf
1024 19:48:05.478222 (+ 8747us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3742394752}
I20251024 19:48:05.494068 2272 consensus_queue.cc:1048] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:05.494622 1767 tablet_bootstrap.cc:492] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.494798 1767 ts_tablet_manager.cc:1403] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.070s user 0.008s sys 0.004s
I20251024 19:48:05.496186 1767 raft_consensus.cc:359] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.496420 1767 raft_consensus.cc:385] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.496511 1767 raft_consensus.cc:740] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.496819 1767 consensus_queue.cc:260] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.497637 1767 ts_tablet_manager.cc:1434] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251024 19:48:05.498034 1767 tablet_bootstrap.cc:492] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.499511 1649 raft_consensus.cc:3060] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.501444 1767 tablet_bootstrap.cc:654] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.513302 1767 tablet_bootstrap.cc:492] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.513607 1767 ts_tablet_manager.cc:1403] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20251024 19:48:05.514019 1476 tablet_service.cc:1505] Processing CreateTablet for tablet a0a6a1cd6e314c20b55d857c1564d188 (DEFAULT_TABLE table=default.Table_415e7ca23c72439a9bebab742ea81150 [id=91564a7da2b147a2a682b31b67cf7ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.514523 1767 raft_consensus.cc:359] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.514853 1767 raft_consensus.cc:385] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.514981 1767 raft_consensus.cc:740] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.515158 1340 tablet_service.cc:1505] Processing CreateTablet for tablet a0a6a1cd6e314c20b55d857c1564d188 (DEFAULT_TABLE table=default.Table_415e7ca23c72439a9bebab742ea81150 [id=91564a7da2b147a2a682b31b67cf7ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.515637 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0a6a1cd6e314c20b55d857c1564d188. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.515373 1767 consensus_queue.cc:260] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.516481 1767 ts_tablet_manager.cc:1434] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251024 19:48:05.516842 1767 tablet_bootstrap.cc:492] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.518779 1767 tablet_bootstrap.cc:654] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.519399 1488 tablet_service.cc:1505] Processing CreateTablet for tablet fa9c5b8254d3427f907879af997ac112 (DEFAULT_TABLE table=default.Table_fc16485e80804cc990dbd6f6aca14502 [id=2072a7f448f54ba0bc79a0ed009a5fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.519975 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 95ae017a2e084ccabfa69c9a5befe0e7 (DEFAULT_TABLE table=default.Table_c367e7fcf06a48fcbc2311c7a4998cac [id=3310cfdc71e7494c941fe1b6abe5f39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.520215 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 157f84b56067415da404df30fc8a5755 (DEFAULT_TABLE table=default.Table_29c99dcba4ab4430aa0b7cb3c70621e9 [id=f03496d30031410d924792109e4bbf5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.520431 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 157f84b56067415da404df30fc8a5755 (DEFAULT_TABLE table=default.Table_29c99dcba4ab4430aa0b7cb3c70621e9 [id=f03496d30031410d924792109e4bbf5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.520709 1768 tablet_bootstrap.cc:492] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:05.521051 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 95ae017a2e084ccabfa69c9a5befe0e7 (DEFAULT_TABLE table=default.Table_c367e7fcf06a48fcbc2311c7a4998cac [id=3310cfdc71e7494c941fe1b6abe5f39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.521497 1624 tablet_service.cc:1505] Processing CreateTablet for tablet fa9c5b8254d3427f907879af997ac112 (DEFAULT_TABLE table=default.Table_fc16485e80804cc990dbd6f6aca14502 [id=2072a7f448f54ba0bc79a0ed009a5fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.521920 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 95ae017a2e084ccabfa69c9a5befe0e7 (DEFAULT_TABLE table=default.Table_c367e7fcf06a48fcbc2311c7a4998cac [id=3310cfdc71e7494c941fe1b6abe5f39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.522548 1336 tablet_service.cc:1505] Processing CreateTablet for tablet fa9c5b8254d3427f907879af997ac112 (DEFAULT_TABLE table=default.Table_fc16485e80804cc990dbd6f6aca14502 [id=2072a7f448f54ba0bc79a0ed009a5fe0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:05.524528 1444 consensus_peers.cc:597] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31647a48fb834c80b6920c756a94250d. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:05.529320 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 15de0d8aa9e545c09233186bbe361b97
I20251024 19:48:05.531006 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 15de0d8aa9e545c09233186bbe361b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.532904 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 15de0d8aa9e545c09233186bbe361b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:05.533228 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 15de0d8aa9e545c09233186bbe361b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:05.534570 1767 tablet_bootstrap.cc:492] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.534782 1767 ts_tablet_manager.cc:1403] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.005s
I20251024 19:48:05.535702 1767 raft_consensus.cc:359] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.535940 1767 raft_consensus.cc:385] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.535952 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fb1b7f0a2371441380ca5492cf1c6a3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:05.536079 1767 raft_consensus.cc:740] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.536332 1767 consensus_queue.cc:260] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:05.536556 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 1d04a9b54bd6446d8deb64292cef672d (table default.Table_147e888401aa40e5a60d97884b36cb98 [id=1acdc5d0bfb94e0faf47055b20d803b0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1d04a9b54bd6446d8deb64292cef672d consensus state has no leader
I20251024 19:48:05.537001 1767 ts_tablet_manager.cc:1434] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:05.537289 1767 tablet_bootstrap.cc:492] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:48:05.537303 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb1b7f0a2371441380ca5492cf1c6a3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.538772 1508 raft_consensus.cc:2468] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:05.538993 1767 tablet_bootstrap.cc:654] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.539229 1650 raft_consensus.cc:2468] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:05.539853 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0a6a1cd6e314c20b55d857c1564d188. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.540426 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157f84b56067415da404df30fc8a5755. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.540987 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ae017a2e084ccabfa69c9a5befe0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.541435 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa9c5b8254d3427f907879af997ac112. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.541739 1308 leader_election.cc:304] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
W20251024 19:48:05.544756 1485 consensus_meta.cc:287] T 74029a7d7cdc4503a833c1758887b684 P 10a9536207e343f188e0ef709081d04e: Time spent flushing consensus metadata: real 0.732s user 0.002s sys 0.005s
I20251024 19:48:05.545717 2272 raft_consensus.cc:2804] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:05.545873 2272 raft_consensus.cc:697] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:05.546180 2272 consensus_queue.cc:237] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.551081 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 030e690ce4df4818a6450ae293b0ffbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:05.560571 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3e6f8764e9ae472b9ac35a032182aed1
I20251024 19:48:05.562402 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e6f8764e9ae472b9ac35a032182aed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.565508 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e6f8764e9ae472b9ac35a032182aed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:05.565784 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e6f8764e9ae472b9ac35a032182aed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:05.566630 1768 tablet_bootstrap.cc:654] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.570325 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 1d04a9b54bd6446d8deb64292cef672d (table default.Table_147e888401aa40e5a60d97884b36cb98 [id=1acdc5d0bfb94e0faf47055b20d803b0]). Expected version 1 got 0
I20251024 19:48:05.572708 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a1f1892bd3b4776adc7c2519fcdf411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.573247 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet dc1b6f0a377945089c50012f74d38564 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ca8f818440f4ae7a63c0901fcdcdf59) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.573841 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 030e690ce4df4818a6450ae293b0ffbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.575809 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0947a6b976074a85abef51497e84b52e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.576821 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0947a6b976074a85abef51497e84b52e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.583201 1775 ts_tablet_manager.cc:1929] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:05.583962 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157f84b56067415da404df30fc8a5755. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.584648 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ae017a2e084ccabfa69c9a5befe0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.584601 2246 raft_consensus.cc:493] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.584825 2246 raft_consensus.cc:515] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.585116 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa9c5b8254d3427f907879af997ac112. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.585511 2246 leader_election.cc:290] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:05.586304 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d04a9b54bd6446d8deb64292cef672d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:05.586545 1371 raft_consensus.cc:2468] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:05.587018 1586 leader_election.cc:304] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:05.587637 2246 raft_consensus.cc:2804] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:05.587742 2246 raft_consensus.cc:493] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:05.587857 2246 raft_consensus.cc:3060] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.603524 1774 ts_tablet_manager.cc:1929] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:05.603699 1774 log.cc:1199] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/2496f103436047dea5dabb577651249f
I20251024 19:48:05.604161 1774 ts_tablet_manager.cc:1950] T 2496f103436047dea5dabb577651249f P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:05.607678 1775 log.cc:1199] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/2496f103436047dea5dabb577651249f
I20251024 19:48:05.607874 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ae017a2e084ccabfa69c9a5befe0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.608584 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa9c5b8254d3427f907879af997ac112. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.612447 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3fd8aa1d9aca429dbfe46f472ebbf53b
I20251024 19:48:05.613065 1184 catalog_manager.cc:6719] ReplaceTablet: tablet 3fd8aa1d9aca429dbfe46f472ebbf53b deleted and replaced by tablet 6bebcdb80476423b8e6c750f749a1771
I20251024 19:48:05.617017 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 505c30f0b2b749ce918b578903db3757
I20251024 19:48:05.617944 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_36fd0e9f25584f9293267ff08c700997" } new_table_name: "default.Table_dfeadb429d564640af3d8d78fa50e64e" modify_external_catalogs: true
I20251024 19:48:05.620990 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b with delete_type TABLET_DATA_DELETED (replaced by tablet 6bebcdb80476423b8e6c750f749a1771) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.622341 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 44ee03ede02c4bdab706fde910978332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.622691 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 505c30f0b2b749ce918b578903db3757 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.623116 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet e3b4e85f741849b2a6a3e8574396ff88 with delete_type TABLET_DATA_DELETED (replaced by tablet 769f576810ab49368eb0fc7837b7da58) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.623138 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ea7e190995344cb8a1c35c95fc35b820" } new_table_name: "default.Table_6bb9ce85a6e341e7b2cc6d9ffc2c0127" modify_external_catalogs: true
I20251024 19:48:05.623826 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b with delete_type TABLET_DATA_DELETED (replaced by tablet 6bebcdb80476423b8e6c750f749a1771) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:05.627341 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 157f84b56067415da404df30fc8a5755 (DEFAULT_TABLE table=default.Table_29c99dcba4ab4430aa0b7cb3c70621e9 [id=f03496d30031410d924792109e4bbf5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.627735 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157f84b56067415da404df30fc8a5755. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.630788 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 68246f886bbc498a9f47830b78a72b3f (DEFAULT_TABLE table=default.Table_5f58bdde786443b5841b7e8d61400808 [id=bebfd2f8b7544b44ad0b9925f24b14d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.631515 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 505c30f0b2b749ce918b578903db3757 (DEFAULT_TABLE table=default.Table_ae2a66ac44f249a2b5335fa64ff1c99c [id=a74d8f22e4f6434195fef6a9ad2b97a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.632157 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 489752031a5a4d168367cb4d143702b7 (DEFAULT_TABLE table=default.Table_2622ff0be0624668b5534e85df77e487 [id=2bc8a33982a245b5b78e72e20a1ab7de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.632869 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 68ca2dd438d94bf0b5f1fc69f0a94517 (DEFAULT_TABLE table=default.Table_c570270b7d7b4f5091c34e0550dd46ef [id=ed879934f5754b92b8cff234f3f66333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.634940 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7c880a4af94d4041b8c7cf2066002ee3
I20251024 19:48:05.636384 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c880a4af94d4041b8c7cf2066002ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.638228 2141 raft_consensus.cc:515] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.638945 2141 leader_election.cc:290] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.639065 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 308) took 3487 ms. Trace:
I20251024 19:48:05.639251 1774 rpcz_store.cc:276] 1024 19:48:02.151743 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.151814 (+ 71us) service_pool.cc:225] Handling call
1024 19:48:05.149939 (+2998125us) ts_tablet_manager.cc:1151] Deleting tablet 2496f103436047dea5dabb577651249f
1024 19:48:05.149949 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.255030 (+105081us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.603378 (+348348us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.638900 (+ 35522us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2997935}
I20251024 19:48:05.639882 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 309) took 3487 ms. Trace:
I20251024 19:48:05.640043 1774 rpcz_store.cc:276] 1024 19:48:02.152702 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.152760 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:05.639549 (+3486789us) ts_tablet_manager.cc:1151] Deleting tablet 96a6e0a4882c42399fd07f06ac21aa3f
1024 19:48:05.639557 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.639720 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3486588}
I20251024 19:48:05.640508 1774 tablet_replica.cc:333] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:05.640798 1774 raft_consensus.cc:2243] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:05.641315 1774 raft_consensus.cc:2272] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:05.642360 1776 ts_tablet_manager.cc:1929] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:05.642514 1776 log.cc:1199] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/9623d7baf0174a019e2353a5effef081
I20251024 19:48:05.646071 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_63e00ebf66c2426a96902b5443eaa366"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.646660 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63e00ebf66c2426a96902b5443eaa366 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.625195 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 505c30f0b2b749ce918b578903db3757 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:05.650194 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d04a9b54bd6446d8deb64292cef672d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:05.650403 1508 raft_consensus.cc:2468] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:05.651196 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "030e690ce4df4818a6450ae293b0ffbd" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:05.651399 1649 raft_consensus.cc:3060] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.652129 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 68246f886bbc498a9f47830b78a72b3f (DEFAULT_TABLE table=default.Table_5f58bdde786443b5841b7e8d61400808 [id=bebfd2f8b7544b44ad0b9925f24b14d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.652240 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 68ca2dd438d94bf0b5f1fc69f0a94517 (DEFAULT_TABLE table=default.Table_c570270b7d7b4f5091c34e0550dd46ef [id=ed879934f5754b92b8cff234f3f66333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.652730 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c880a4af94d4041b8c7cf2066002ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:05.653468 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489752031a5a4d168367cb4d143702b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.653475 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "030e690ce4df4818a6450ae293b0ffbd" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:05.653651 1372 raft_consensus.cc:3060] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.653824 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7da1601a87524ef1a8b5069ba6823843
I20251024 19:48:05.656423 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505c30f0b2b749ce918b578903db3757. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.656989 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c880a4af94d4041b8c7cf2066002ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:05.657109 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 505c30f0b2b749ce918b578903db3757 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:05.657986 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5c78cfb1ef614578accc5ec60079087b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.658593 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c78cfb1ef614578accc5ec60079087b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.658633 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 7da1601a87524ef1a8b5069ba6823843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.661891 1776 ts_tablet_manager.cc:1950] T 9623d7baf0174a019e2353a5effef081 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:05.662750 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 7da1601a87524ef1a8b5069ba6823843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:05.661990 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 505c30f0b2b749ce918b578903db3757 (DEFAULT_TABLE table=default.Table_ae2a66ac44f249a2b5335fa64ff1c99c [id=a74d8f22e4f6434195fef6a9ad2b97a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.664973 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 2496f103436047dea5dabb577651249f (table default.Table_9ecacbf3e0fe41b7b54b20dbd81c4e05 [id=4e52676da82e4f5f807ffc32fb747281]) successfully deleted
I20251024 19:48:05.665177 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505c30f0b2b749ce918b578903db3757. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.666321 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68246f886bbc498a9f47830b78a72b3f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:05.666636 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 96a6e0a4882c42399fd07f06ac21aa3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96a6e0a4882c42399fd07f06ac21aa3f already in progress: creating tablet
I20251024 19:48:05.667104 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68ca2dd438d94bf0b5f1fc69f0a94517. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.667559 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 68ca2dd438d94bf0b5f1fc69f0a94517 (DEFAULT_TABLE table=default.Table_c570270b7d7b4f5091c34e0550dd46ef [id=ed879934f5754b92b8cff234f3f66333]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.669909 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68ca2dd438d94bf0b5f1fc69f0a94517. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.673997 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 68246f886bbc498a9f47830b78a72b3f (DEFAULT_TABLE table=default.Table_5f58bdde786443b5841b7e8d61400808 [id=bebfd2f8b7544b44ad0b9925f24b14d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.677031 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68246f886bbc498a9f47830b78a72b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.678973 2146 raft_consensus.cc:493] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.679194 2146 raft_consensus.cc:515] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.679335 1774 ts_tablet_manager.cc:1916] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:05.681434 2146 leader_election.cc:290] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.682612 2146 raft_consensus.cc:493] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.682957 2146 raft_consensus.cc:515] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.683890 2146 leader_election.cc:290] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.684432 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f346a90a9efe442f82750336e3893deb" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:05.684672 1370 raft_consensus.cc:2468] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:05.684803 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81f8393f78204860bd854f4872099387" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:05.685278 1371 raft_consensus.cc:2393] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
I20251024 19:48:05.443879 1960 rpcz_store.cc:276] 1024 19:48:03.804197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.804255 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:03.812410 (+ 8155us) raft_consensus.cc:1471] Updating replica for 180 ops
1024 19:48:03.816663 (+ 4253us) raft_consensus.cc:1534] Early marking committed up to index 0
1024 19:48:03.816669 (+ 6us) raft_consensus.cc:1539] Triggering prepare for 180 ops
1024 19:48:03.816793 (+ 124us) write_op.cc:270] Start()
1024 19:48:03.816831 (+ 38us) write_op.cc:276] Timestamp: P: 1761335280333464 usec, L: 0
1024 19:48:04.236559 (+419728us) write_op.cc:270] Start()
1024 19:48:04.236618 (+ 59us) write_op.cc:276] Timestamp: P: 1761335280335962 usec, L: 0
1024 19:48:04.236864 (+ 246us) write_op.cc:270] Start()
1024 19:48:04.236895 (+ 31us) write_op.cc:276] Timestamp: P: 1761335280417783 usec, L: 0
1024 19:48:04.237036 (+ 141us) write_op.cc:270] Start()
1024 19:48:04.237057 (+ 21us) write_op.cc:276] Timestamp: P: 1761335280431779 usec, L: 0
1024 19:48:04.237178 (+ 121us) write_op.cc:270] Start()
1024 19:48:04.237195 (+ 17us) write_op.cc:276] Timestamp: P: 1761335280437591 usec, L: 0
1024 19:48:04.237308 (+ 113us) write_op.cc:270] Start()
1024 19:48:04.237326 (+ 18us) write_op.cc:276] Timestamp: P: 1761335280443108 usec, L: 0
1024 19:48:04.237477 (+ 151us) write_op.cc:270] Start()
1024 19:48:04.237526 (+ 49us) write_op.cc:276] Timestamp: P: 1761335280453439 usec, L: 0
1024 19:48:04.237689 (+ 163us) write_op.cc:270] Start()
1024 19:48:04.237708 (+ 19us) write_op.cc:276] Timestamp: P: 1761335280464591 usec, L: 0
1024 19:48:04.237853 (+ 145us) write_op.cc:270] Start()
1024 19:48:04.237871 (+ 18us) write_op.cc:276] Timestamp: P: 1761335280488148 usec, L: 0
1024 19:48:04.237993 (+ 122us) write_op.cc:270] Start()
1024 19:48:04.238009 (+ 16us) write_op.cc:276] Timestamp: P: 1761335280579769 usec, L: 0
1024 19:48:04.238117 (+ 108us) write_op.cc:270] Start()
1024 19:48:04.238134 (+ 17us) write_op.cc:276] Timestamp: P: 1761335280582333 usec, L: 0
1024 19:48:04.238257 (+ 123us) write_op.cc:270] Start()
1024 19:48:04.238283 (+ 26us) write_op.cc:276] Timestamp: P: 1761335280584541 usec, L: 0
1024 19:48:04.238433 (+ 150us) write_op.cc:270] Start()
1024 19:48:04.238453 (+ 20us) write_op.cc:276] Timestamp: P: 1761335280586844 usec, L: 0
1024 19:48:04.238605 (+ 152us) write_op.cc:270] Start()
1024 19:48:04.238626 (+ 21us) write_op.cc:276] Timestamp: P: 1761335280589348 usec, L: 0
1024 19:48:04.238801 (+ 175us) write_op.cc:270] Start()
1024 19:48:04.238836 (+ 35us) write_op.cc:276] Timestamp: P: 1761335280591323 usec, L: 0
1024 19:48:04.238976 (+ 140us) write_op.cc:270] Start()
1024 19:48:04.238995 (+ 19us) write_op.cc:276] Timestamp: P: 1761335280593896 usec, L: 0
1024 19:48:04.239155 (+ 160us) write_op.cc:270] Start()
1024 19:48:04.239178 (+ 23us) write_op.cc:276] Timestamp: P: 1761335280610036 usec, L: 0
1024 19:48:04.239313 (+ 135us) write_op.cc:270] Start()
1024 19:48:04.239330 (+ 17us) write_op.cc:276] Timestamp: P: 1761335280716116 usec, L: 0
1024 19:48:04.239469 (+ 139us) write_op.cc:270] Start()
1024 19:48:04.239489 (+ 20us) write_op.cc:276] Timestamp: P: 1761335280758006 usec, L: 0
1024 19:48:04.239612 (+ 123us) write_op.cc:270] Start()
1024 19:48:04.239630 (+ 18us) write_op.cc:276] Timestamp: P: 1761335280798031 usec, L: 0
1024 19:48:04.239780 (+ 150us) write_op.cc:270] Start()
1024 19:48:04.239802 (+ 22us) write_op.cc:276] Timestamp: P: 1761335280855739 usec, L: 0
1024 19:48:04.239960 (+ 158us) write_op.cc:270] Start()
1024 19:48:04.239980 (+ 20us) write_op.cc:276] Timestamp: P: 1761335280892663 usec, L: 0
1024 19:48:04.240104 (+ 124us) write_op.cc:270] Start()
1024 19:48:04.240122 (+ 18us) write_op.cc:276] Timestamp: P: 1761335280931624 usec, L: 0
1024 19:48:04.240269 (+ 147us) write_op.cc:270] Start()
1024 19:48:04.240290 (+ 21us) write_op.cc:276] Timestamp: P: 1761335280941524 usec, L: 0
1024 19:48:04.240438 (+ 148us) write_op.cc:270] Start()
1024 19:48:04.240458 (+ 20us) write_op.cc:276] Timestamp: P: 1761335281025066 usec, L: 0
1024 19:48:04.240595 (+ 137us) write_op.cc:270] Start()
1024 19:48:04.240616 (+ 21us) write_op.cc:276] Timestamp: P: 1761335281025873 usec, L: 0
1024 19:48:04.240794 (+ 178us) write_op.cc:270] Start()
1024 19:48:04.240828 (+ 34us) write_op.cc:276] Timestamp: P: 1761335281026572 usec, L: 0
1024 19:48:04.240946 (+ 118us) write_op.cc:270] Start()
1024 19:48:04.240962 (+ 16us) write_op.cc:276] Timestamp: P: 1761335281027229 usec, L: 0
1024 19:48:04.337478 (+ 96516us) write_op.cc:270] Start()
1024 19:48:04.337624 (+ 146us) write_op.cc:276] Timestamp: P: 1761335281027823 usec, L: 0
1024 19:48:04.337921 (+ 297us) write_op.cc:270] Start()
1024 19:48:04.347116 (+ 9195us) write_op.cc:276] Timestamp: P: 1761335281028372 usec, L: 0
1024 19:48:04.347932 (+ 816us) write_op.cc:270] Start()
1024 19:48:04.347992 (+ 60us) write_op.cc:276] Timestamp: P: 1761335281028930 usec, L: 0
1024 19:48:04.348233 (+ 241us) write_op.cc:270] Start()
1024 19:48:04.348262 (+ 29us) write_op.cc:276] Timestamp: P: 1761335281070279 usec, L: 0
1024 19:48:04.348468 (+ 206us) write_op.cc:270] Start()
1024 19:48:04.348794 (+ 326us) write_op.cc:276] Timestamp: P: 1761335281243196 usec, L: 0
1024 19:48:04.349150 (+ 356us) write_op.cc:270] Start()
1024 19:48:04.349178 (+ 28us) write_op.cc:276] Timestamp: P: 1761335281269240 usec, L: 0
1024 19:48:04.349349 (+ 171us) write_op.cc:270] Start()
1024 19:48:04.349374 (+ 25us) write_op.cc:276] Timestamp: P: 1761335281270682 usec, L: 0
1024 19:48:04.349511 (+ 137us) write_op.cc:270] Start()
1024 19:48:04.349532 (+ 21us) write_op.cc:276] Timestamp: P: 1761335281304637 usec, L: 0
1024 19:48:04.349700 (+ 168us) write_op.cc:270] Start()
1024 19:48:04.349731 (+ 31us) write_op.cc:276] Timestamp: P: 1761335281305304 usec, L: 0
1024 19:48:04.349863 (+ 132us) write_op.cc:270] Start()
1024 19:48:04.349904 (+ 41us) write_op.cc:276] Timestamp: P: 1761335281306025 usec, L: 0
1024 19:48:04.350040 (+ 136us) write_op.cc:270] Start()
1024 19:48:04.350058 (+ 18us) write_op.cc:276] Timestamp: P: 1761335281308013 usec, L: 0
1024 19:48:04.350186 (+ 128us) write_op.cc:270] Start()
1024 19:48:04.350204 (+ 18us) write_op.cc:276] Timestamp: P: 1761335281308922 usec, L: 0
1024 19:48:04.350331 (+ 127us) write_op.cc:270] Start()
1024 19:48:04.350371 (+ 40us) write_op.cc:276] Timestamp: P: 1761335281309958 usec, L: 0
1024 19:48:04.350544 (+ 173us) write_op.cc:270] Start()
1024 19:48:04.350563 (+ 19us) write_op.cc:276] Timestamp: P: 1761335281450345 usec, L: 0
1024 19:48:04.350736 (+ 173us) write_op.cc:270] Start()
1024 19:48:04.350757 (+ 21us) write_op.cc:276] Timestamp: P: 1761335281450994 usec, L: 0
1024 19:48:04.351088 (+ 331us) write_op.cc:270] Start()
1024 19:48:04.351138 (+ 50us) write_op.cc:276] Timestamp: P: 1761335281451548 usec, L: 0
1024 19:48:04.351356 (+ 218us) write_op.cc:270] Start()
1024 19:48:04.351382 (+ 26us) write_op.cc:276] Timestamp: P: 1761335281489272 usec, L: 0
1024 19:48:04.406385 (+ 55003us) write_op.cc:270] Start()
1024 19:48:04.406429 (+ 44us) write_op.cc:276] Timestamp: P: 1761335281503947 usec, L: 0
1024 19:48:04.423892 (+ 17463us) write_op.cc:270] Start()
1024 19:48:04.423930 (+ 38us) write_op.cc:276] Timestamp: P: 1761335281504621 usec, L: 0
1024 19:48:04.424134 (+ 204us) write_op.cc:270] Start()
1024 19:48:04.424160 (+ 26us) write_op.cc:276] Timestamp: P: 1761335281506516 usec, L: 0
1024 19:48:04.424296 (+ 136us) write_op.cc:270] Start()
1024 19:48:04.424318 (+ 22us) write_op.cc:276] Timestamp: P: 1761335281507075 usec, L: 0
1024 19:48:04.424454 (+ 136us) write_op.cc:270] Start()
1024 19:48:04.424476 (+ 22us) write_op.cc:276] Timestamp: P: 1761335281507588 usec, L: 0
1024 19:48:04.424610 (+ 134us) write_op.cc:270] Start()
1024 19:48:04.424629 (+ 19us) write_op.cc:276] Timestamp: P: 1761335281508500 usec, L: 0
1024 19:48:04.424760 (+ 131us) write_op.cc:270] Start()
1024 19:48:04.424783 (+ 23us) write_op.cc:276] Timestamp: P: 1761335281568231 usec, L: 0
1024 19:48:05.293968 (+869185us) write_op.cc:270] Start()
1024 19:48:05.294011 (+ 43us) write_op.cc:276] Timestamp: P: 1761335281578844 usec, L: 0
1024 19:48:05.299763 (+ 5752us) spinlock_profiling.cc:243] Waited 5.48 ms on lock 0x616000060778. stack: 00007f81a26cd9f6 00007f81a26cd7b0 00007f81a16a93ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f81af9577d2 00007f81af945daa 00007f81af8f16ff 00007f81af8f07be 00007f81ae5575c1 00007f81ae576d08 00007f81ae574607 00007f81b23e9d8d 00007f81ac58ab46 00007f81ac58a875
1024 19:48:05.299974 (+ 211us) write_op.cc:270] Start()
1024 19:48:05.300008 (+ 34us) write_op.cc:276] Timestamp: P: 1761335281589321 usec, L: 0
1024 19:48:05.300200 (+ 192us) write_op.cc:270] Start()
1024 19:48:05.300221 (+ 21us) write_op.cc:276] Timestamp: P: 1761335281603381 usec, L: 0
1024 19:48:05.300353 (+ 132us) write_op.cc:270] Start()
1024 19:48:05.300372 (+ 19us) write_op.cc:276] Timestamp: P: 1761335281615961 usec, L: 0
1024 19:48:05.300550 (+ 178us) write_op.cc:270] Start()
1024 19:48:05.300589 (+ 39us) write_op.cc:276] Timestamp: P: 1761335281624199 usec, L: 0
1024 19:48:05.300768 (+ 179us) write_op.cc:270] Start()
1024 19:48:05.300797 (+ 29us) write_op.cc:276] Timestamp: P: 1761335281638808 usec, L: 0
1024 19:48:05.300967 (+ 170us) write_op.cc:270] Start()
1024 19:48:05.300988 (+ 21us) write_op.cc:276] Timestamp: P: 1761335281657381 usec, L: 0
1024 19:48:05.301118 (+ 130us) write_op.cc:270] Start()
1024 19:48:05.301143 (+ 25us) write_op.cc:276] Timestamp: P: 1761335281669820 usec, L: 0
1024 19:48:05.301334 (+ 191us) write_op.cc:270] Start()
1024 19:48:05.301354 (+ 20us) write_op.cc:276] Timestamp: P: 1761335281708352 usec, L: 0
1024 19:48:05.301512 (+ 158us) write_op.cc:270] Start()
1024 19:48:05.301536 (+ 24us) write_op.cc:276] Timestamp: P: 1761335281725334 usec, L: 0
1024 19:48:05.301707 (+ 171us) write_op.cc:270] Start()
1024 19:48:05.301727 (+ 20us) write_op.cc:276] Timestamp: P: 1761335281735621 usec, L: 0
1024 19:48:05.301865 (+ 138us) write_op.cc:270] Start()
1024 19:48:05.301887 (+ 22us) write_op.cc:276] Timestamp: P: 1761335281748104 usec, L: 0
1024 19:48:05.302044 (+ 157us) write_op.cc:270] Start()
1024 19:48:05.302065 (+ 21us) write_op.cc:276] Timestamp: P: 1761335281771779 usec, L: 0
1024 19:48:05.302209 (+ 144us) write_op.cc:270] Start()
1024 19:48:05.302230 (+ 21us) write_op.cc:276] Timestamp: P: 1761335281786635 usec, L: 0
1024 19:48:05.302374 (+ 144us) write_op.cc:270] Start()
1024 19:48:05.302395 (+ 21us) write_op.cc:276] Timestamp: P: 1761335281798989 usec, L: 0
1024 19:48:05.302533 (+ 138us) write_op.cc:270] Start()
1024 19:48:05.302554 (+ 21us) write_op.cc:276] Timestamp: P: 1761335281813371 usec, L: 0
1024 19:48:05.302693 (+ 139us) write_op.cc:270] Start()
1024 19:48:05.302715 (+ 22us) write_op.cc:276] Timestamp: P: 1761335281825825 usec, L: 0
1024 19:48:05.302855 (+ 140us) write_op.cc:270] Start()
1024 19:48:05.302874 (+ 19us) write_op.cc:276] Timestamp: P: 1761335281839216 usec, L: 0
1024 19:48:05.303052 (+ 178us) write_op.cc:270] Start()
1024 19:48:05.303077 (+ 25us) write_op.cc:276] Timestamp: P: 1761335281852564 usec, L: 0
1024 19:48:05.303219 (+ 142us) write_op.cc:270] Start()
1024 19:48:05.303238 (+ 19us) write_op.cc:276] Timestamp: P: 1761335281867381 usec, L: 0
1024 19:48:05.305559 (+ 2321us) write_op.cc:270] Start()
1024 19:48:05.305616 (+ 57us) write_op.cc:276] Timestamp: P: 1761335281880647 usec, L: 0
1024 19:48:05.305769 (+ 153us) write_op.cc:270] Start()
1024 19:48:05.305792 (+ 23us) write_op.cc:276] Timestamp: P: 1761335281900709 usec, L: 0
1024 19:48:05.305961 (+ 169us) write_op.cc:270] Start()
1024 19:48:05.305999 (+ 38us) write_op.cc:276] Timestamp: P: 1761335281921309 usec, L: 0
1024 19:48:05.306129 (+ 130us) write_op.cc:270] Start()
1024 19:48:05.306148 (+ 19us) write_op.cc:276] Timestamp: P: 1761335281931872 usec, L: 0
1024 19:48:05.306323 (+ 175us) write_op.cc:270] Start()
1024 19:48:05.306344 (+ 21us) write_op.cc:276] Timestamp: P: 1761335281948116 usec, L: 0
1024 19:48:05.306551 (+ 207us) write_op.cc:270] Start()
1024 19:48:05.306574 (+ 23us) write_op.cc:276] Timestamp: P: 1761335281959307 usec, L: 0
1024 19:48:05.306720 (+ 146us) write_op.cc:270] Start()
1024 19:48:05.306741 (+ 21us) write_op.cc:276] Timestamp: P: 1761335281967902 usec, L: 0
1024 19:48:05.306923 (+ 182us) write_op.cc:270] Start()
1024 19:48:05.306948 (+ 25us) write_op.cc:276] Timestamp: P: 1761335281976647 usec, L: 0
1024 19:48:05.307087 (+ 139us) write_op.cc:270] Start()
1024 19:48:05.307106 (+ 19us) write_op.cc:276] Timestamp: P: 1761335281991110 usec, L: 0
1024 19:48:05.307261 (+ 155us) write_op.cc:270] Start()
1024 19:48:05.307281 (+ 20us) write_op.cc:276] Timestamp: P: 1761335281996364 usec, L: 0
1024 19:48:05.307432 (+ 151us) write_op.cc:270] Start()
1024 19:48:05.307452 (+ 20us) write_op.cc:276] Timestamp: P: 1761335282048982 usec, L: 0
1024 19:48:05.307607 (+ 155us) write_op.cc:270] Start()
1024 19:48:05.307626 (+ 19us) write_op.cc:276] Timestamp: P: 1761335282080401 usec, L: 0
1024 19:48:05.307998 (+ 372us) write_op.cc:270] Start()
1024 19:48:05.308023 (+ 25us) write_op.cc:276] Timestamp: P: 1761335282096788 usec, L: 0
1024 19:48:05.308163 (+ 140us) write_op.cc:270] Start()
1024 19:48:05.308183 (+ 20us) write_op.cc:276] Timestamp: P: 1761335282119733 usec, L: 0
1024 19:48:05.308302 (+ 119us) write_op.cc:270] Start()
1024 19:48:05.308323 (+ 21us) write_op.cc:276] Timestamp: P: 1761335282134932 usec, L: 0
1024 19:48:05.308468 (+ 145us) write_op.cc:270] Start()
1024 19:48:05.308497 (+ 29us) write_op.cc:276] Timestamp: P: 1761335282136570 usec, L: 0
1024 19:48:05.308656 (+ 159us) write_op.cc:270] Start()
1024 19:48:05.308677 (+ 21us) write_op.cc:276] Timestamp: P: 1761335282137415 usec, L: 0
1024 19:48:05.308814 (+ 137us) write_op.cc:270] Start()
1024 19:48:05.308834 (+ 20us) write_op.cc:276] Timestamp: P: 1761335282138262 usec, L: 0
1024 19:48:05.308962 (+ 128us) write_op.cc:270] Start()
1024 19:48:05.308984 (+ 22us) write_op.cc:276] Timestamp: P: 1761335282139050 usec, L: 0
1024 19:48:05.309132 (+ 148us) write_op.cc:270] Start()
1024 19:48:05.309155 (+ 23us) write_op.cc:276] Timestamp: P: 1761335282202996 usec, L: 0
1024 19:48:05.309369 (+ 214us) write_op.cc:270] Start()
1024 19:48:05.309394 (+ 25us) write_op.cc:276] Timestamp: P: 1761335282243201 usec, L: 0
1024 19:48:05.309587 (+ 193us) write_op.cc:270] Start()
1024 19:48:05.309610 (+ 23us) write_op.cc:276] Timestamp: P: 1761335282261281 usec, L: 0
1024 19:48:05.309726 (+ 116us) write_op.cc:270] Start()
1024 19:48:05.309747 (+ 21us) write_op.cc:276] Timestamp: P: 1761335282269057 usec, L: 0
1024 19:48:05.309909 (+ 162us) write_op.cc:270] Start()
1024 19:48:05.309929 (+ 20us) write_op.cc:276] Timestamp: P: 1761335282289464 usec, L: 0
1024 19:48:05.310122 (+ 193us) write_op.cc:270] Start()
1024 19:48:05.310144 (+ 22us) write_op.cc:276] Timestamp: P: 1761335282294559 usec, L: 0
1024 19:48:05.310277 (+ 133us) write_op.cc:270] Start()
1024 19:48:05.310302 (+ 25us) write_op.cc:276] Timestamp: P: 1761335282319587 usec, L: 0
1024 19:48:05.310622 (+ 320us) write_op.cc:270] Start()
1024 19:48:05.310645 (+ 23us) write_op.cc:276] Timestamp: P: 1761335282333423 usec, L: 0
1024 19:48:05.310801 (+ 156us) write_op.cc:270] Start()
1024 19:48:05.310824 (+ 23us) write_op.cc:276] Timestamp: P: 1761335282350266 usec, L: 0
1024 19:48:05.310961 (+ 137us) write_op.cc:270] Start()
1024 19:48:05.310982 (+ 21us) write_op.cc:276] Timestamp: P: 1761335282359261 usec, L: 0
1024 19:48:05.311099 (+ 117us) write_op.cc:270] Start()
1024 19:48:05.311115 (+ 16us) write_op.cc:276] Timestamp: P: 1761335282369182 usec, L: 0
1024 19:48:05.311246 (+ 131us) write_op.cc:270] Start()
1024 19:48:05.311263 (+ 17us) write_op.cc:276] Timestamp: P: 1761335282372460 usec, L: 0
1024 19:48:05.311443 (+ 180us) write_op.cc:270] Start()
1024 19:48:05.311467 (+ 24us) write_op.cc:276] Timestamp: P: 1761335282375920 usec, L: 0
1024 19:48:05.311655 (+ 188us) write_op.cc:270] Start()
1024 19:48:05.311678 (+ 23us) write_op.cc:276] Timestamp: P: 1761335282388964 usec, L: 0
1024 19:48:05.311858 (+ 180us) write_op.cc:270] Start()
1024 19:48:05.311879 (+ 21us) write_op.cc:276] Timestamp: P: 1761335282423411 usec, L: 0
1024 19:48:05.312016 (+ 137us) write_op.cc:270] Start()
1024 19:48:05.312034 (+ 18us) write_op.cc:276] Timestamp: P: 1761335282458512 usec, L: 0
1024 19:48:05.312197 (+ 163us) write_op.cc:270] Start()
1024 19:48:05.312216 (+ 19us) write_op.cc:276] Timestamp: P: 1761335282474114 usec, L: 0
1024 19:48:05.312368 (+ 152us) write_op.cc:270] Start()
1024 19:48:05.312392 (+ 24us) write_op.cc:276] Timestamp: P: 1761335282486240 usec, L: 0
1024 19:48:05.312542 (+ 150us) write_op.cc:270] Start()
1024 19:48:05.312562 (+ 20us) write_op.cc:276] Timestamp: P: 1761335282487550 usec, L: 0
1024 19:48:05.312690 (+ 128us) write_op.cc:270] Start()
1024 19:48:05.312707 (+ 17us) write_op.cc:276] Timestamp: P: 1761335282501327 usec, L: 0
1024 19:48:05.312828 (+ 121us) write_op.cc:270] Start()
1024 19:48:05.312846 (+ 18us) write_op.cc:276] Timestamp: P: 1761335282506532 usec, L: 0
1024 19:48:05.312976 (+ 130us) write_op.cc:270] Start()
1024 19:48:05.312994 (+ 18us) write_op.cc:276] Timestamp: P: 1761335282507177 usec, L: 0
1024 19:48:05.313110 (+ 116us) write_op.cc:270] Start()
1024 19:48:05.313130 (+ 20us) write_op.cc:276] Timestamp: P: 1761335282507724 usec, L: 0
1024 19:48:05.314512 (+ 1382us) write_op.cc:270] Start()
1024 19:48:05.314538 (+ 26us) write_op.cc:276] Timestamp: P: 1761335282508249 usec, L: 0
1024 19:48:05.314749 (+ 211us) write_op.cc:270] Start()
1024 19:48:05.314775 (+ 26us) write_op.cc:276] Timestamp: P: 1761335282508748 usec, L: 0
1024 19:48:05.314975 (+ 200us) write_op.cc:270] Start()
1024 19:48:05.314999 (+ 24us) write_op.cc:276] Timestamp: P: 1761335282572071 usec, L: 0
1024 19:48:05.315126 (+ 127us) write_op.cc:270] Start()
1024 19:48:05.315144 (+ 18us) write_op.cc:276] Timestamp: P: 1761335282585204 usec, L: 0
1024 19:48:05.315281 (+ 137us) write_op.cc:270] Start()
1024 19:48:05.315305 (+ 24us) write_op.cc:276] Timestamp: P: 1761335282620501 usec, L: 0
1024 19:48:05.323779 (+ 8474us) write_op.cc:270] Start()
1024 19:48:05.323822 (+ 43us) write_op.cc:276] Timestamp: P: 1761335282630248 usec, L: 0
1024 19:48:05.324244 (+ 422us) write_op.cc:270] Start()
1024 19:48:05.324275 (+ 31us) write_op.cc:276] Timestamp: P: 1761335282649476 usec, L: 0
1024 19:48:05.324628 (+ 353us) write_op.cc:270] Start()
1024 19:48:05.324654 (+ 26us) write_op.cc:276] Timestamp: P: 1761335282651263 usec, L: 0
1024 19:48:05.324851 (+ 197us) write_op.cc:270] Start()
1024 19:48:05.324876 (+ 25us) write_op.cc:276] Timestamp: P: 1761335282651917 usec, L: 0
1024 19:48:05.325018 (+ 142us) write_op.cc:270] Start()
1024 19:48:05.325038 (+ 20us) write_op.cc:276] Timestamp: P: 1761335282652496 usec, L: 0
1024 19:48:05.325216 (+ 178us) write_op.cc:270] Start()
1024 19:48:05.325235 (+ 19us) write_op.cc:276] Timestamp: P: 1761335282653039 usec, L: 0
1024 19:48:05.325487 (+ 252us) write_op.cc:270] Start()
1024 19:48:05.325514 (+ 27us) write_op.cc:276] Timestamp: P: 1761335282654818 usec, L: 0
1024 19:48:05.325684 (+ 170us) write_op.cc:270] Start()
1024 19:48:05.325710 (+ 26us) write_op.cc:276] Timestamp: P: 1761335282655333 usec, L: 0
1024 19:48:05.325940 (+ 230us) write_op.cc:270] Start()
1024 19:48:05.326010 (+ 70us) write_op.cc:276] Timestamp: P: 1761335282657846 usec, L: 0
1024 19:48:05.326319 (+ 309us) write_op.cc:270] Start()
1024 19:48:05.326351 (+ 32us) write_op.cc:276] Timestamp: P: 1761335282700851 usec, L: 0
1024 19:48:05.326585 (+ 234us) write_op.cc:270] Start()
1024 19:48:05.326613 (+ 28us) write_op.cc:276] Timestamp: P: 1761335282817176 usec, L: 0
1024 19:48:05.326790 (+ 177us) write_op.cc:270] Start()
1024 19:48:05.326812 (+ 22us) write_op.cc:276] Timestamp: P: 1761335282857590 usec, L: 0
1024 19:48:05.327036 (+ 224us) write_op.cc:270] Start()
1024 19:48:05.327059 (+ 23us) write_op.cc:276] Timestamp: P: 1761335282861059 usec, L: 0
1024 19:48:05.327252 (+ 193us) write_op.cc:270] Start()
1024 19:48:05.327277 (+ 25us) write_op.cc:276] Timestamp: P: 1761335282864064 usec, L: 0
1024 19:48:05.327448 (+ 171us) write_op.cc:270] Start()
1024 19:48:05.327478 (+ 30us) write_op.cc:276] Timestamp: P: 1761335282866540 usec, L: 0
1024 19:48:05.327642 (+ 164us) write_op.cc:270] Start()
1024 19:48:05.327662 (+ 20us) write_op.cc:276] Timestamp: P: 1761335282869045 usec, L: 0
1024 19:48:05.327886 (+ 224us) write_op.cc:270] Start()
1024 19:48:05.327909 (+ 23us) write_op.cc:276] Timestamp: P: 1761335282994768 usec, L: 0
1024 19:48:05.328082 (+ 173us) write_op.cc:270] Start()
1024 19:48:05.328103 (+ 21us) write_op.cc:276] Timestamp: P: 1761335282995365 usec, L: 0
1024 19:48:05.328271 (+ 168us) write_op.cc:270] Start()
1024 19:48:05.328291 (+ 20us) write_op.cc:276] Timestamp: P: 1761335282996025 usec, L: 0
1024 19:48:05.328456 (+ 165us) write_op.cc:270] Start()
1024 19:48:05.328486 (+ 30us) write_op.cc:276] Timestamp: P: 1761335282996518 usec, L: 0
1024 19:48:05.328655 (+ 169us) write_op.cc:270] Start()
1024 19:48:05.328678 (+ 23us) write_op.cc:276] Timestamp: P: 1761335283008884 usec, L: 0
1024 19:48:05.328867 (+ 189us) write_op.cc:270] Start()
1024 19:48:05.328893 (+ 26us) write_op.cc:276] Timestamp: P: 1761335283009524 usec, L: 0
1024 19:48:05.329038 (+ 145us) write_op.cc:270] Start()
1024 19:48:05.329059 (+ 21us) write_op.cc:276] Timestamp: P: 1761335283010073 usec, L: 0
1024 19:48:05.329195 (+ 136us) write_op.cc:270] Start()
1024 19:48:05.329211 (+ 16us) write_op.cc:276] Timestamp: P: 1761335283068716 usec, L: 0
1024 19:48:05.329356 (+ 145us) write_op.cc:270] Start()
1024 19:48:05.329375 (+ 19us) write_op.cc:276] Timestamp: P: 1761335283069505 usec, L: 0
1024 19:48:05.329525 (+ 150us) write_op.cc:270] Start()
1024 19:48:05.329545 (+ 20us) write_op.cc:276] Timestamp: P: 1761335283070116 usec, L: 0
1024 19:48:05.339066 (+ 9521us) write_op.cc:270] Start()
1024 19:48:05.339101 (+ 35us) write_op.cc:276] Timestamp: P: 1761335283070651 usec, L: 0
1024 19:48:05.339272 (+ 171us) write_op.cc:270] Start()
1024 19:48:05.339293 (+ 21us) write_op.cc:276] Timestamp: P: 1761335283071206 usec, L: 0
1024 19:48:05.339493 (+ 200us) write_op.cc:270] Start()
1024 19:48:05.339516 (+ 23us) write_op.cc:276] Timestamp: P: 1761335283125117 usec, L: 0
1024 19:48:05.339671 (+ 155us) write_op.cc:270] Start()
1024 19:48:05.339696 (+ 25us) write_op.cc:276] Timestamp: P: 1761335283127377 usec, L: 0
1024 19:48:05.339845 (+ 149us) write_op.cc:270] Start()
1024 19:48:05.339869 (+ 24us) write_op.cc:276] Timestamp: P: 1761335283207819 usec, L: 0
1024 19:48:05.340044 (+ 175us) write_op.cc:270] Start()
1024 19:48:05.340065 (+ 21us) write_op.cc:276] Timestamp: P: 1761335283209323 usec, L: 0
1024 19:48:05.340208 (+ 143us) write_op.cc:270] Start()
1024 19:48:05.340230 (+ 22us) write_op.cc:276] Timestamp: P: 1761335283273826 usec, L: 0
1024 19:48:05.340391 (+ 161us) write_op.cc:270] Start()
1024 19:48:05.340414 (+ 23us) write_op.cc:276] Timestamp: P: 1761335283309977 usec, L: 0
1024 19:48:05.340566 (+ 152us) write_op.cc:270] Start()
1024 19:48:05.340590 (+ 24us) write_op.cc:276] Timestamp: P: 1761335283326334 usec, L: 0
1024 19:48:05.340756 (+ 166us) write_op.cc:270] Start()
1024 19:48:05.340778 (+ 22us) write_op.cc:276] Timestamp: P: 1761335283328633 usec, L: 0
1024 19:48:05.340969 (+ 191us) write_op.cc:270] Start()
1024 19:48:05.340990 (+ 21us) write_op.cc:276] Timestamp: P: 1761335283375367 usec, L: 0
1024 19:48:05.341167 (+ 177us) write_op.cc:270] Start()
1024 19:48:05.341191 (+ 24us) write_op.cc:276] Timestamp: P: 1761335283402191 usec, L: 0
1024 19:48:05.341378 (+ 187us) write_op.cc:270] Start()
1024 19:48:05.341398 (+ 20us) write_op.cc:276] Timestamp: P: 1761335283404693 usec, L: 0
1024 19:48:05.341532 (+ 134us) write_op.cc:270] Start()
1024 19:48:05.341551 (+ 19us) write_op.cc:276] Timestamp: P: 1761335283438425 usec, L: 0
1024 19:48:05.341711 (+ 160us) write_op.cc:270] Start()
1024 19:48:05.341732 (+ 21us) write_op.cc:276] Timestamp: P: 1761335283448873 usec, L: 0
1024 19:48:05.341878 (+ 146us) write_op.cc:270] Start()
1024 19:48:05.341903 (+ 25us) write_op.cc:276] Timestamp: P: 1761335283476350 usec, L: 0
1024 19:48:05.342042 (+ 139us) write_op.cc:270] Start()
1024 19:48:05.342066 (+ 24us) write_op.cc:276] Timestamp: P: 1761335283489957 usec, L: 0
1024 19:48:05.342226 (+ 160us) write_op.cc:270] Start()
1024 19:48:05.342246 (+ 20us) write_op.cc:276] Timestamp: P: 1761335283513142 usec, L: 0
1024 19:48:05.342435 (+ 189us) write_op.cc:270] Start()
1024 19:48:05.342456 (+ 21us) write_op.cc:276] Timestamp: P: 1761335283532903 usec, L: 0
1024 19:48:05.342635 (+ 179us) write_op.cc:270] Start()
1024 19:48:05.342659 (+ 24us) write_op.cc:276] Timestamp: P: 1761335283576388 usec, L: 0
1024 19:48:05.342846 (+ 187us) write_op.cc:270] Start()
1024 19:48:05.342872 (+ 26us) write_op.cc:276] Timestamp: P: 1761335283635663 usec, L: 0
1024 19:48:05.343017 (+ 145us) write_op.cc:270] Start()
1024 19:48:05.343038 (+ 21us) write_op.cc:276] Timestamp: P: 1761335283689610 usec, L: 0
1024 19:48:05.343187 (+ 149us) write_op.cc:270] Start()
1024 19:48:05.343208 (+ 21us) write_op.cc:276] Timestamp: P: 1761335283708565 usec, L: 0
1024 19:48:05.343343 (+ 135us) write_op.cc:270] Start()
1024 19:48:05.343360 (+ 17us) write_op.cc:276] Timestamp: P: 1761335283725808 usec, L: 0
1024 19:48:05.343506 (+ 146us) write_op.cc:270] Start()
1024 19:48:05.343524 (+ 18us) write_op.cc:276] Timestamp: P: 1761335283747497 usec, L: 0
1024 19:48:05.343683 (+ 159us) write_op.cc:270] Start()
1024 19:48:05.343704 (+ 21us) write_op.cc:276] Timestamp: P: 1761335283752048 usec, L: 0
1024 19:48:05.343837 (+ 133us) write_op.cc:270] Start()
1024 19:48:05.343859 (+ 22us) write_op.cc:276] Timestamp: P: 1761335283753009 usec, L: 0
1024 19:48:05.344010 (+ 151us) write_op.cc:270] Start()
1024 19:48:05.344032 (+ 22us) write_op.cc:276] Timestamp: P: 1761335283753900 usec, L: 0
1024 19:48:05.344182 (+ 150us) write_op.cc:270] Start()
1024 19:48:05.344202 (+ 20us) write_op.cc:276] Timestamp: P: 1761335283754829 usec, L: 0
1024 19:48:05.344359 (+ 157us) write_op.cc:270] Start()
1024 19:48:05.344379 (+ 20us) write_op.cc:276] Timestamp: P: 1761335283755809 usec, L: 0
1024 19:48:05.344521 (+ 142us) write_op.cc:270] Start()
1024 19:48:05.344543 (+ 22us) write_op.cc:276] Timestamp: P: 1761335283756593 usec, L: 0
1024 19:48:05.344688 (+ 145us) write_op.cc:270] Start()
1024 19:48:05.344707 (+ 19us) write_op.cc:276] Timestamp: P: 1761335283757422 usec, L: 0
1024 19:48:05.344845 (+ 138us) write_op.cc:270] Start()
1024 19:48:05.344864 (+ 19us) write_op.cc:276] Timestamp: P: 1761335283758418 usec, L: 0
1024 19:48:05.344981 (+ 117us) write_op.cc:270] Start()
1024 19:48:05.344997 (+ 16us) write_op.cc:276] Timestamp: P: 1761335283759203 usec, L: 0
1024 19:48:05.345152 (+ 155us) write_op.cc:270] Start()
1024 19:48:05.345170 (+ 18us) write_op.cc:276] Timestamp: P: 1761335283792502 usec, L: 0
1024 19:48:05.358145 (+ 12975us) log.cc:844] Serialized 116074 byte log entry
1024 19:48:05.368610 (+ 10465us) raft_consensus.cc:1645] Marking committed up to 514
1024 19:48:05.376316 (+ 7706us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:05.376320 (+ 4us) write_op.cc:454] Released schema lock
1024 19:48:05.383526 (+ 7206us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:05.383535 (+ 9us) write_op.cc:454] Released schema lock
1024 19:48:05.387070 (+ 3535us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:05.387075 (+ 5us) write_op.cc:454] Released schema lock
1024 19:48:05.391354 (+ 4279us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:05.391360 (+ 6us) write_op.cc:454] Released schema lock
1024 19:48:05.435239 (+ 43879us) raft_consensus.cc:1697] Filling consensus response to leader.
1024 19:48:05.440982 (+ 5743us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1024 19:48:05.440993 (+ 11us) raft_consensus.cc:1685] finished
1024 19:48:05.440995 (+ 2us) raft_consensus.cc:1691] UpdateReplicas() finished
1024 19:48:05.441080 (+ 85us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1024 19:48:04.235802 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1024 19:48:04.235973 (+ 171us) write_op.cc:432] Acquiring schema lock in shared mode
1024 19:48:04.235981 (+ 8us) write_op.cc:435] Acquired schema lock
1024 19:48:04.235984 (+ 3us) tablet.cc:660] Decoding operations
1024 19:48:04.236112 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
1024 19:48:04.236132 (+ 20us) write_op.cc:641] Partition lock acquired for write op
1024 19:48:04.236137 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1024 19:48:04.236231 (+ 94us) tablet.cc:699] Row locks acquired
1024 19:48:04.236235 (+ 4us) write_op.cc:260] PREPARE: finished
1024 19:48:05.368689 (+1132454us) op_driver.cc:464] REPLICATION: finished
1024 19:48:05.375263 (+ 6574us) write_op.cc:301] APPLY: starting
1024 19:48:05.375286
I20251024 19:48:05.685626 1586 leader_election.cc:304] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:05.688585 2146 raft_consensus.cc:2804] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251024 19:48:05.688717 2146 raft_consensus.cc:493] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:05.689422 2146 raft_consensus.cc:3060] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Advancing to term 2
I20251024 19:48:05.689778 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 505c30f0b2b749ce918b578903db3757 (DEFAULT_TABLE table=default.Table_ae2a66ac44f249a2b5335fa64ff1c99c [id=a74d8f22e4f6434195fef6a9ad2b97a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.691689 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68246f886bbc498a9f47830b78a72b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.701892 1768 tablet_bootstrap.cc:492] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:05.702201 1768 ts_tablet_manager.cc:1403] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.183s user 0.001s sys 0.003s
I20251024 19:48:05.703027 1768 raft_consensus.cc:359] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.703241 1768 raft_consensus.cc:385] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.703339 1768 raft_consensus.cc:740] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.703609 1768 consensus_queue.cc:260] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.705615 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b with delete_type TABLET_DATA_DELETED (replaced by tablet 6bebcdb80476423b8e6c750f749a1771) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:05.657306 2173 consensus_queue.cc:1048] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.042s
I20251024 19:48:05.707208 2271 consensus_queue.cc:1048] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:05.708307 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505c30f0b2b749ce918b578903db3757. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.713020 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 7da1601a87524ef1a8b5069ba6823843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:05.714828 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f346a90a9efe442f82750336e3893deb" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:05.715330 1775 ts_tablet_manager.cc:1950] T 2496f103436047dea5dabb577651249f P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:05.716621 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68ca2dd438d94bf0b5f1fc69f0a94517. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.718516 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 340) took 3566 ms. Trace:
I20251024 19:48:05.718631 2141 consensus_queue.cc:1048] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20251024 19:48:05.719038 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81f8393f78204860bd854f4872099387" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:05.719854 1586 leader_election.cc:304] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:05.721925 1768 ts_tablet_manager.cc:1434] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20251024 19:48:05.724997 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 406) took 2283 ms. Trace:
I20251024 19:48:05.725119 1776 rpcz_store.cc:276] 1024 19:48:03.441957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.442014 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:05.178473 (+1736459us) ts_tablet_manager.cc:1151] Deleting tablet 9623d7baf0174a019e2353a5effef081
1024 19:48:05.178487 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.604413 (+425926us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.642216 (+ 37803us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.724939 (+ 82723us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1736237}
I20251024 19:48:05.725641 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 409) took 2216 ms. Trace:
I20251024 19:48:05.725734 1776 rpcz_store.cc:276] 1024 19:48:03.509454 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.509522 (+ 68us) service_pool.cc:225] Handling call
1024 19:48:05.725453 (+2215931us) ts_tablet_manager.cc:1151] Deleting tablet 38818c7997c84cedad5f9b09eef77105
1024 19:48:05.725464 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.725567 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2215503}
I20251024 19:48:05.725782 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 489752031a5a4d168367cb4d143702b7 (DEFAULT_TABLE table=default.Table_2622ff0be0624668b5534e85df77e487 [id=2bc8a33982a245b5b78e72e20a1ab7de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.726109 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 413) took 2145 ms. Trace:
I20251024 19:48:05.726197 1776 rpcz_store.cc:276] 1024 19:48:03.580384 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.580437 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:05.725949 (+2145512us) ts_tablet_manager.cc:1151] Deleting tablet 701c93a600964c20a275f5942a2ff863
1024 19:48:05.725956 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.726057 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2139138}
I20251024 19:48:05.726487 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 414) took 2139 ms. Trace:
I20251024 19:48:05.726553 1776 rpcz_store.cc:276] 1024 19:48:03.587070 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.588845 (+ 1775us) service_pool.cc:225] Handling call
1024 19:48:05.726362 (+2137517us) ts_tablet_manager.cc:1151] Deleting tablet 786fe2fd6fca407e83a76f74f96a8b16
1024 19:48:05.726369 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.726444 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2136385}
I20251024 19:48:05.726861 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 415) took 2090 ms. Trace:
I20251024 19:48:05.726928 1776 rpcz_store.cc:276] 1024 19:48:03.635991 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.636129 (+ 138us) service_pool.cc:225] Handling call
1024 19:48:05.726749 (+2090620us) ts_tablet_manager.cc:1151] Deleting tablet ee3c521876324808857f7724028c895f
1024 19:48:05.726757 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.726820 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2089890}
I20251024 19:48:05.727238 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 416) took 2080 ms. Trace:
I20251024 19:48:05.727342 1776 rpcz_store.cc:276] 1024 19:48:03.646514 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.646573 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:05.727088 (+2080515us) ts_tablet_manager.cc:1151] Deleting tablet ee68243b4a664575b679950603bb250e
1024 19:48:05.727095 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.727172 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2080280}
I20251024 19:48:05.727753 1776 tablet_replica.cc:333] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:05.728024 1776 raft_consensus.cc:2243] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:05.728281 1776 raft_consensus.cc:2272] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:05.728451 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 505e4fb0755a4caaa462e5f5154bfee2
I20251024 19:48:05.728994 1192 catalog_manager.cc:6719] ReplaceTablet: tablet 505e4fb0755a4caaa462e5f5154bfee2 deleted and replaced by tablet 4b39e014607b4437b255c26c92c71861
I20251024 19:48:05.729252 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet 505e4fb0755a4caaa462e5f5154bfee2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b39e014607b4437b255c26c92c71861) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:05.732535 1776 ts_tablet_manager.cc:1916] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:05.737008 2321 raft_consensus.cc:3060] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.737149 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dd245479047544d68efcf1ec83acc6b8" } new_table_name: "default.Table_1c5fc102a45e49948a8157aa0b5e6949" modify_external_catalogs: true
I20251024 19:48:05.737221 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet aebf76d4a66947ed9756643740418798 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:05.738993 1192 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 1d04a9b54bd6446d8deb64292cef672d (table default.Table_147e888401aa40e5a60d97884b36cb98 [id=1acdc5d0bfb94e0faf47055b20d803b0]). Expected version 1 got 0
I20251024 19:48:05.739567 1192 catalog_manager.cc:5649] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:05.740442 1192 catalog_manager.cc:5649] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:05.740898 1192 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 6a7e76a3cac443ec8d2ab466075aae6b (table default.Table_e63d3e693e1148a7b8d2b5731b1027bc [id=5fc5a169166b436ba9343a652a650b8e]). Expected version 1 got 0
I20251024 19:48:05.741230 1192 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 037c8cc1e53447109fafc047c02f196f (table default.Table_e87be853efba49aeb50e71270beac600 [id=5bc052b3e0f04699b4b391bcec33617d]). Expected version 1 got 0
I20251024 19:48:05.741645 1192 catalog_manager.cc:5649] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:05.742291 1372 raft_consensus.cc:2468] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:05.742295 2246 raft_consensus.cc:515] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.743005 2246 leader_election.cc:290] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:05.743242 2246 raft_consensus.cc:493] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.743505 2246 raft_consensus.cc:515] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.744134 2246 leader_election.cc:290] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.744371 2246 raft_consensus.cc:493] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.744545 2246 raft_consensus.cc:515] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.747378 1192 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]). Expected version 1 got 0
I20251024 19:48:05.753846 2246 leader_election.cc:290] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.762135 1768 tablet_bootstrap.cc:492] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:05.762104 2020 raft_consensus.cc:493] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.762295 2020 raft_consensus.cc:515] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.762930 2020 leader_election.cc:290] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.763617 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3b4e85f741849b2a6a3e8574396ff88" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:05.763880 1648 raft_consensus.cc:2468] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:05.764945 1648 raft_consensus.cc:3060] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.765807 1775 rpcz_store.cc:276] 1024 19:48:02.151800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.151860 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:05.365417 (+3213557us) ts_tablet_manager.cc:1151] Deleting tablet 2496f103436047dea5dabb577651249f
1024 19:48:05.365428 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.462452 (+ 97024us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.583039 (+120587us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.718185 (+135146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3212756}
I20251024 19:48:05.767784 1447 leader_election.cc:304] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:05.770298 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 2496f103436047dea5dabb577651249f (table default.Table_9ecacbf3e0fe41b7b54b20dbd81c4e05 [id=4e52676da82e4f5f807ffc32fb747281]) successfully deleted
I20251024 19:48:05.770543 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 9623d7baf0174a019e2353a5effef081 (table default.Table_7f14fdc529ec48d68d03ca041f64a53f [id=281060b712614bbeb46a006b6e34a91e]) successfully deleted
W20251024 19:48:05.770841 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 38818c7997c84cedad5f9b09eef77105 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38818c7997c84cedad5f9b09eef77105 already in progress: creating tablet
I20251024 19:48:05.770925 1444 leader_election.cc:304] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:05.771185 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 786fe2fd6fca407e83a76f74f96a8b16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 786fe2fd6fca407e83a76f74f96a8b16 already in progress: creating tablet
W20251024 19:48:05.771441 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet ee3c521876324808857f7724028c895f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee3c521876324808857f7724028c895f already in progress: creating tablet
W20251024 19:48:05.771711 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet ee68243b4a664575b679950603bb250e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee68243b4a664575b679950603bb250e already in progress: creating tablet
I20251024 19:48:05.772241 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 505e4fb0755a4caaa462e5f5154bfee2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b39e014607b4437b255c26c92c71861) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:05.773025 1618 tablet_service.cc:1505] Processing CreateTablet for tablet eee675874516489f9e8dac41f70ac4e4 (DEFAULT_TABLE table=default.Table_aec0055f05e645e5a429d88125c4fb6f [id=1c7ef3d892274e26ad80000b1925db2d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:05.776854 1447 leader_election.cc:343] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: e3b4e85f741849b2a6a3e8574396ff88
I20251024 19:48:05.777217 2141 raft_consensus.cc:2804] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:05.777482 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 505e4fb0755a4caaa462e5f5154bfee2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b39e014607b4437b255c26c92c71861) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:05.777631 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8df56cc2917245cca143934fa22702f5" } new_table_name: "default.Table_10657b3692ee4b7081e10b2ce7e87b07" modify_external_catalogs: true
I20251024 19:48:05.777782 1766 tablet_bootstrap.cc:492] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:05.777961 1766 ts_tablet_manager.cc:1403] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.442s user 0.040s sys 0.027s
I20251024 19:48:05.778342 2173 raft_consensus.cc:493] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:05.778702 2173 raft_consensus.cc:515] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.779556 2173 leader_election.cc:290] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:05.780818 2141 raft_consensus.cc:697] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:05.780871 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc1b6f0a377945089c50012f74d38564" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:05.781095 1372 raft_consensus.cc:2468] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:05.781124 2141 consensus_queue.cc:237] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.781724 1447 leader_election.cc:304] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:05.781770 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 341) took 3623 ms. Trace:
I20251024 19:48:05.781868 1775 rpcz_store.cc:276] 1024 19:48:02.153247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.153299 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:05.768646 (+3615347us) ts_tablet_manager.cc:1151] Deleting tablet 96a6e0a4882c42399fd07f06ac21aa3f
1024 19:48:05.768668 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.776723 (+ 8055us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3615144}
I20251024 19:48:05.782219 1775 tablet_replica.cc:333] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:05.782413 1775 raft_consensus.cc:2243] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:05.782801 1775 raft_consensus.cc:2272] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:05.783442 1775 ts_tablet_manager.cc:1916] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:05.783828 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d04a9b54bd6446d8deb64292cef672d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:05.783948 1340 tablet_service.cc:1505] Processing CreateTablet for tablet eee675874516489f9e8dac41f70ac4e4 (DEFAULT_TABLE table=default.Table_aec0055f05e645e5a429d88125c4fb6f [id=1c7ef3d892274e26ad80000b1925db2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.784040 1372 raft_consensus.cc:3060] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.784267 2020 raft_consensus.cc:2804] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:05.784364 2020 raft_consensus.cc:493] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:05.784555 2020 raft_consensus.cc:3060] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.785841 2224 raft_consensus.cc:2804] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:05.785992 2224 raft_consensus.cc:493] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:05.786198 2224 raft_consensus.cc:3060] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.787591 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc1b6f0a377945089c50012f74d38564" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:05.788156 1647 raft_consensus.cc:2468] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:05.788396 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d04a9b54bd6446d8deb64292cef672d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:05.788596 1510 raft_consensus.cc:3060] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.789750 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3b4e85f741849b2a6a3e8574396ff88" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:05.790714 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 96a6e0a4882c42399fd07f06ac21aa3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96a6e0a4882c42399fd07f06ac21aa3f already in progress: creating tablet
I20251024 19:48:05.791531 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 489752031a5a4d168367cb4d143702b7 (DEFAULT_TABLE table=default.Table_2622ff0be0624668b5534e85df77e487 [id=2bc8a33982a245b5b78e72e20a1ab7de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.791895 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489752031a5a4d168367cb4d143702b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.793764 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913534ab88dd459e9cac2faebb5492a1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:05.794746 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913534ab88dd459e9cac2faebb5492a1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:05.795022 1370 raft_consensus.cc:2393] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
W20251024 19:48:05.796700 1586 leader_election.cc:343] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: e3b4e85f741849b2a6a3e8574396ff88
I20251024 19:48:05.797046 1586 leader_election.cc:304] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:05.797843 2323 raft_consensus.cc:3060] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.799959 2322 consensus_queue.cc:1048] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:05.801977 1766 raft_consensus.cc:359] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.802206 1766 raft_consensus.cc:385] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.802310 1766 raft_consensus.cc:740] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.802560 1766 consensus_queue.cc:260] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.803071 1766 ts_tablet_manager.cc:1434] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
I20251024 19:48:05.803440 1766 tablet_bootstrap.cc:492] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:05.804147 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c30ba0fd50f2482793bf9e2caecd448e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.804811 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c30ba0fd50f2482793bf9e2caecd448e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.807916 1624 tablet_service.cc:1505] Processing CreateTablet for tablet c89fa8932dbb4f4fbf7fd9f57bff27c3 (DEFAULT_TABLE table=default.Table_6b41a041a50a4f2ca796ea21033f1ab9 [id=4f2ab360359e4e14ac8cacf8c063f5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.809617 1488 tablet_service.cc:1505] Processing CreateTablet for tablet c89fa8932dbb4f4fbf7fd9f57bff27c3 (DEFAULT_TABLE table=default.Table_6b41a041a50a4f2ca796ea21033f1ab9 [id=4f2ab360359e4e14ac8cacf8c063f5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.810097 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c89fa8932dbb4f4fbf7fd9f57bff27c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.791363 1474 tablet_service.cc:1505] Processing CreateTablet for tablet eee675874516489f9e8dac41f70ac4e4 (DEFAULT_TABLE table=default.Table_aec0055f05e645e5a429d88125c4fb6f [id=1c7ef3d892274e26ad80000b1925db2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.814837 1344 tablet_service.cc:1505] Processing CreateTablet for tablet c89fa8932dbb4f4fbf7fd9f57bff27c3 (DEFAULT_TABLE table=default.Table_6b41a041a50a4f2ca796ea21033f1ab9 [id=4f2ab360359e4e14ac8cacf8c063f5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.816382 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 7c880a4af94d4041b8c7cf2066002ee3 (DEFAULT_TABLE table=default.Table_61b7f6c428aa4bd9a65c1d66171c59c9 [id=6c604c6f86f944a285866376d9a5ce65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.821959 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 7c880a4af94d4041b8c7cf2066002ee3 (DEFAULT_TABLE table=default.Table_61b7f6c428aa4bd9a65c1d66171c59c9 [id=6c604c6f86f944a285866376d9a5ce65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.825654 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 41176885447e42409ee4363bb4d903f2 (DEFAULT_TABLE table=default.Table_8ae396f2397d425097a8f71cb1c9c910 [id=be1cc9d047554ea6a9afa531fb08a81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.827288 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 41176885447e42409ee4363bb4d903f2 (DEFAULT_TABLE table=default.Table_8ae396f2397d425097a8f71cb1c9c910 [id=be1cc9d047554ea6a9afa531fb08a81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.827579 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 7c880a4af94d4041b8c7cf2066002ee3 (DEFAULT_TABLE table=default.Table_61b7f6c428aa4bd9a65c1d66171c59c9 [id=6c604c6f86f944a285866376d9a5ce65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.828998 1011 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskc9NInd/build/asan/bin/kudu with pid 1825
I20251024 19:48:05.830976 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 41176885447e42409ee4363bb4d903f2 (DEFAULT_TABLE table=default.Table_8ae396f2397d425097a8f71cb1c9c910 [id=be1cc9d047554ea6a9afa531fb08a81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.831291 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 46e5e391a7674b5d8e1dd1a780fe36c8 (DEFAULT_TABLE table=default.Table_8df56cc2917245cca143934fa22702f5 [id=e40de65cb381461bb0e5c9896a5eb812]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.832121 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 46e5e391a7674b5d8e1dd1a780fe36c8 (DEFAULT_TABLE table=default.Table_8df56cc2917245cca143934fa22702f5 [id=e40de65cb381461bb0e5c9896a5eb812]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.832929 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 46e5e391a7674b5d8e1dd1a780fe36c8 (DEFAULT_TABLE table=default.Table_8df56cc2917245cca143934fa22702f5 [id=e40de65cb381461bb0e5c9896a5eb812]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:05.841332 1767 tablet_bootstrap.cc:492] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.841598 1767 ts_tablet_manager.cc:1403] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.304s user 0.012s sys 0.019s
I20251024 19:48:05.842479 1767 raft_consensus.cc:359] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.842716 1767 raft_consensus.cc:385] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.842801 1767 raft_consensus.cc:740] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.843365 1767 consensus_queue.cc:260] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.843969 1767 ts_tablet_manager.cc:1434] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251024 19:48:05.854418 1767 tablet_bootstrap.cc:492] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.866358 1768 tablet_bootstrap.cc:654] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:05.866358 1743 log.cc:927] Time spent T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20251024 19:48:05.866436 1766 tablet_bootstrap.cc:654] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:05.866681 1767 tablet_bootstrap.cc:654] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:05.867192 1737 log.cc:927] Time spent T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20251024 19:48:05.867736 1774 ts_tablet_manager.cc:1929] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:05.867851 1774 log.cc:1199] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/f05ad092936b4bafa3ae795904c2df78
I20251024 19:48:05.868378 1774 ts_tablet_manager.cc:1950] T f05ad092936b4bafa3ae795904c2df78 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:05.869484 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489752031a5a4d168367cb4d143702b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.869815 2224 raft_consensus.cc:515] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.870103 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee675874516489f9e8dac41f70ac4e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.870579 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c89fa8932dbb4f4fbf7fd9f57bff27c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.870579 2224 leader_election.cc:290] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.870925 1510 raft_consensus.cc:2468] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:05.871040 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c880a4af94d4041b8c7cf2066002ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.871256 1508 raft_consensus.cc:2393] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
I20251024 19:48:05.871464 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41176885447e42409ee4363bb4d903f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.871961 1649 raft_consensus.cc:2468] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:05.872560 1372 raft_consensus.cc:2468] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:05.869403 2020 raft_consensus.cc:515] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.874027 2020 leader_election.cc:290] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:05.874949 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc1b6f0a377945089c50012f74d38564" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:05.875119 1647 raft_consensus.cc:3060] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.876925 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e5e391a7674b5d8e1dd1a780fe36c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.878283 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3b4e85f741849b2a6a3e8574396ff88" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:05.878468 1649 raft_consensus.cc:3060] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.880322 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee675874516489f9e8dac41f70ac4e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.881261 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc1b6f0a377945089c50012f74d38564" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:05.881542 1369 raft_consensus.cc:3060] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.886186 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c880a4af94d4041b8c7cf2066002ee3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:05.886266 1447 leader_election.cc:343] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: e3b4e85f741849b2a6a3e8574396ff88
I20251024 19:48:05.900082 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_099958da5fc9487db3398646189a7194"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.900769 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_099958da5fc9487db3398646189a7194 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.905884 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 310) took 3671 ms. Trace:
I20251024 19:48:05.906028 1774 rpcz_store.cc:276] 1024 19:48:02.233960 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.234021 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:05.640319 (+3406298us) ts_tablet_manager.cc:1151] Deleting tablet f05ad092936b4bafa3ae795904c2df78
1024 19:48:05.640327 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.742729 (+102402us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.867577 (+124848us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.905837 (+ 38260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3406070}
W20251024 19:48:05.906219 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 250f106664be4b5aaa330dbb4e43cb7b consensus state has no leader
I20251024 19:48:05.906518 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 320) took 3432 ms. Trace:
I20251024 19:48:05.906589 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet 505e4fb0755a4caaa462e5f5154bfee2 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:05.906621 1774 rpcz_store.cc:276] 1024 19:48:02.473830 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.475880 (+ 2050us) service_pool.cc:225] Handling call
1024 19:48:05.906363 (+3430483us) ts_tablet_manager.cc:1151] Deleting tablet 5d129bdc876e425aabcf1c99997e319a
1024 19:48:05.906375 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.906485 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3430214}
I20251024 19:48:05.907037 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 321) took 3431 ms. Trace:
W20251024 19:48:05.907068 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e38ef2ac19d14f9c91fa8abe86cc7987 consensus state has no leader
I20251024 19:48:05.907162 1774 rpcz_store.cc:276] 1024 19:48:02.476004 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.476179 (+ 175us) service_pool.cc:225] Handling call
1024 19:48:05.906886 (+3430707us) ts_tablet_manager.cc:1151] Deleting tablet f3d80885fcc642b8a75da30cd4abecbd
1024 19:48:05.906899 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.906988 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3430485}
I20251024 19:48:05.907505 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 322) took 3431 ms. Trace:
I20251024 19:48:05.907629 1774 rpcz_store.cc:276] 1024 19:48:02.476147 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.489816 (+ 13669us) service_pool.cc:225] Handling call
1024 19:48:05.907374 (+3417558us) ts_tablet_manager.cc:1151] Deleting tablet 129debb55f4a4b7295a8ad88e55ebc83
1024 19:48:05.907383 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.907462 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3417288}
I20251024 19:48:05.908027 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 326) took 3355 ms. Trace:
I20251024 19:48:05.908154 1774 rpcz_store.cc:276] 1024 19:48:02.552719 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.553636 (+ 917us) service_pool.cc:225] Handling call
1024 19:48:05.907876 (+3354240us) ts_tablet_manager.cc:1151] Deleting tablet 607ae453221d422e9d07882830c02275
1024 19:48:05.907886 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.907996 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3332803}
I20251024 19:48:05.908499 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 327) took 3355 ms. Trace:
I20251024 19:48:05.908604 1774 rpcz_store.cc:276] 1024 19:48:02.552853 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.553659 (+ 806us) service_pool.cc:225] Handling call
1024 19:48:05.908388 (+3354729us) ts_tablet_manager.cc:1151] Deleting tablet d9bf08ccf7054fedb9efcbb00354c71a
1024 19:48:05.908397 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.908468 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3324894}
I20251024 19:48:05.908984 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 329) took 3355 ms. Trace:
I20251024 19:48:05.910167 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet f05ad092936b4bafa3ae795904c2df78 (table default.Table_99584fc98c7e4f0f853914cc3dda2b12 [id=0704c52aae2c421789a94e4391b9ed2a]) successfully deleted
W20251024 19:48:05.910502 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 5d129bdc876e425aabcf1c99997e319a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d129bdc876e425aabcf1c99997e319a already in progress: creating tablet
W20251024 19:48:05.910732 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet f3d80885fcc642b8a75da30cd4abecbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3d80885fcc642b8a75da30cd4abecbd already in progress: creating tablet
W20251024 19:48:05.911001 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 129debb55f4a4b7295a8ad88e55ebc83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 129debb55f4a4b7295a8ad88e55ebc83 already in progress: creating tablet
W20251024 19:48:05.911204 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 607ae453221d422e9d07882830c02275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607ae453221d422e9d07882830c02275 already in progress: creating tablet
W20251024 19:48:05.911657 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet d9bf08ccf7054fedb9efcbb00354c71a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9bf08ccf7054fedb9efcbb00354c71a already in progress: creating tablet
I20251024 19:48:05.909124 1774 rpcz_store.cc:276] 1024 19:48:02.553052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.553848 (+ 796us) service_pool.cc:225] Handling call
1024 19:48:05.908853 (+3355005us) ts_tablet_manager.cc:1151] Deleting tablet 80c58e2439dc490c824b5f0023b5434b
1024 19:48:05.908862 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.908952 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3320466}
I20251024 19:48:05.913901 1369 raft_consensus.cc:2468] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
W20251024 19:48:05.917742 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 80c58e2439dc490c824b5f0023b5434b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80c58e2439dc490c824b5f0023b5434b already in progress: creating tablet
I20251024 19:48:05.919102 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41176885447e42409ee4363bb4d903f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.922639 1447 leader_election.cc:304] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:05.924108 2020 raft_consensus.cc:2804] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
W20251024 19:48:05.924185 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7503b22d9e12481b9af96385b9d76e50 consensus state has no leader
I20251024 19:48:05.924285 2020 raft_consensus.cc:697] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:05.925065 2020 consensus_queue.cc:237] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.930927 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e5e391a7674b5d8e1dd1a780fe36c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.934350 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 94b35252e63f4a8ba305cc029ddb5ae8 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:05.937860 2146 raft_consensus.cc:515] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.938886 1647 raft_consensus.cc:2468] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:05.942087 2146 leader_election.cc:290] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 2 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:05.937536 1649 raft_consensus.cc:2468] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:05.942742 1444 leader_election.cc:304] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:05.943187 2224 raft_consensus.cc:2804] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:05.943385 2224 raft_consensus.cc:697] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:05.943655 2224 consensus_queue.cc:237] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:05.943807 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46e5e391a7674b5d8e1dd1a780fe36c8 consensus state has no leader
I20251024 19:48:05.946992 1650 raft_consensus.cc:1240] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 2 FOLLOWER]: Rejecting Update request from peer 10a9536207e343f188e0ef709081d04e for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20251024 19:48:05.947271 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c8e3fdadfb964ad6b614c60e5d2695c5" } modify_external_catalogs: true
I20251024 19:48:05.947438 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c8e3fdadfb964ad6b614c60e5d2695c5" } modify_external_catalogs: true
I20251024 19:48:05.948397 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e8f91be03f4e46d4b0847a86c58ffc6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.948993 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8f91be03f4e46d4b0847a86c58ffc6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.950707 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e6ec362938cc478d9b9abee36f34f5c5" } modify_external_catalogs: true
I20251024 19:48:05.950861 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e6ec362938cc478d9b9abee36f34f5c5" } modify_external_catalogs: true
I20251024 19:48:05.952080 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_325e96b0d32e4a3aab388a916c29ec26" } modify_external_catalogs: true
I20251024 19:48:05.952219 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_325e96b0d32e4a3aab388a916c29ec26" } modify_external_catalogs: true
I20251024 19:48:05.952757 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7966ff9879d24254aed39562b2f127d7" } modify_external_catalogs: true
I20251024 19:48:05.952903 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7966ff9879d24254aed39562b2f127d7" } modify_external_catalogs: true
I20251024 19:48:05.952946 2173 consensus_queue.cc:1048] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:05.953433 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2f543df6b1194a7195ecd656134e4b17" } modify_external_catalogs: true
I20251024 19:48:05.953616 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2f543df6b1194a7195ecd656134e4b17" } modify_external_catalogs: true
I20251024 19:48:05.954241 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_72be3d9212c64e6dbe03acba1f12df24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:05.954746 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72be3d9212c64e6dbe03acba1f12df24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:05.956151 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_998040d863404075b6422120ab4ce290" } new_table_name: "default.Table_d7e131f8c5f8432d8c00ffbf1857560e" modify_external_catalogs: true
I20251024 19:48:05.956976 2173 consensus_queue.cc:1059] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:05.967417 1586 leader_election.cc:304] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:05.968379 1586 leader_election.cc:304] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:05.979691 1767 tablet_bootstrap.cc:492] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:05.969388 2146 raft_consensus.cc:2804] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:05.979871 2146 raft_consensus.cc:697] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:05.980247 2146 consensus_queue.cc:237] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:05.980320 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f346a90a9efe442f82750336e3893deb" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:05.980439 2246 raft_consensus.cc:2749] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:05.970140 2323 raft_consensus.cc:2749] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:05.977967 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee675874516489f9e8dac41f70ac4e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.982259 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41176885447e42409ee4363bb4d903f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.982483 1648 raft_consensus.cc:3060] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:05.982640 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e5e391a7674b5d8e1dd1a780fe36c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.984046 1767 ts_tablet_manager.cc:1403] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.130s user 0.004s sys 0.002s
I20251024 19:48:05.985322 1767 raft_consensus.cc:359] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.969165 2321 raft_consensus.cc:2749] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:05.985787 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c880a4af94d4041b8c7cf2066002ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.987111 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c89fa8932dbb4f4fbf7fd9f57bff27c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:05.993410 1472 tablet_service.cc:1552] Processing DeleteTablet for tablet 030e690ce4df4818a6450ae293b0ffbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:05.993613 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 2496f103436047dea5dabb577651249f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:05.993831 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 44ee03ede02c4bdab706fde910978332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:05.994062 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet dc1b6f0a377945089c50012f74d38564 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ca8f818440f4ae7a63c0901fcdcdf59) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:05.994262 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a1f1892bd3b4776adc7c2519fcdf411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:57 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:05.994953 1767 raft_consensus.cc:385] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:05.995247 1767 raft_consensus.cc:740] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:05.995626 1767 consensus_queue.cc:260] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:05.996798 1767 ts_tablet_manager.cc:1434] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251024 19:48:05.997610 1767 tablet_bootstrap.cc:492] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:05.999486 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 338) took 3393 ms. Trace:
I20251024 19:48:05.978384 1776 ts_tablet_manager.cc:1929] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:05.999918 1776 log.cc:1199] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/e20881e400cf4fac914b70afc79c084f
I20251024 19:48:06.000371 1776 ts_tablet_manager.cc:1950] T e20881e400cf4fac914b70afc79c084f P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:06.000469 1774 rpcz_store.cc:276] 1024 19:48:02.597067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.597129 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:05.918891 (+3321762us) ts_tablet_manager.cc:1151] Deleting tablet b79801594e95434cbb8fe7b75b14602e
1024 19:48:05.918905 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.990887 (+ 71982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3313753}
I20251024 19:48:06.000917 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 347) took 3297 ms. Trace:
I20251024 19:48:06.001044 1774 rpcz_store.cc:276] 1024 19:48:02.703726 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.703813 (+ 87us) service_pool.cc:225] Handling call
1024 19:48:06.000742 (+3296929us) ts_tablet_manager.cc:1151] Deleting tablet 4a69c0918d2e4eed9d91b1913ec2c21e
1024 19:48:06.000752 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.000882 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3289448}
I20251024 19:48:06.001209 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 417) took 2315 ms. Trace:
I20251024 19:48:06.001403 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 348) took 3295 ms. Trace:
I20251024 19:48:06.001359 1776 rpcz_store.cc:276] 1024 19:48:03.686125 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.686214 (+ 89us) service_pool.cc:225] Handling call
1024 19:48:05.727616 (+2041402us) ts_tablet_manager.cc:1151] Deleting tablet e20881e400cf4fac914b70afc79c084f
1024 19:48:05.727627 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.755646 (+ 28019us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:05.978041 (+222395us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.001157 (+ 23116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2041119}
I20251024 19:48:06.001530 1774 rpcz_store.cc:276] 1024 19:48:02.706229 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.706305 (+ 76us) service_pool.cc:225] Handling call
1024 19:48:06.001251 (+3294946us) ts_tablet_manager.cc:1151] Deleting tablet 2e7b3288182744969e217d4f174fe4ea
1024 19:48:06.001259 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.001371 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3252838}
I20251024 19:48:06.001647 1775 ts_tablet_manager.cc:1929] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.001786 1775 log.cc:1199] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/369cebfda6ee43638b79f8652fa55855
I20251024 19:48:06.001792 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 418) took 2283 ms. Trace:
I20251024 19:48:06.001863 1774 tablet_replica.cc:333] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:06.001888 1776 rpcz_store.cc:276] 1024 19:48:03.718472 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.718528 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:06.001643 (+2283115us) ts_tablet_manager.cc:1151] Deleting tablet 45cff9bcc4924e0081518c284f074940
1024 19:48:06.001652 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.001758 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2282880}
I20251024 19:48:06.002112 1774 raft_consensus.cc:2243] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.002172 1776 tablet_replica.cc:333] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:06.002308 1774 raft_consensus.cc:2272] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.002362 1776 raft_consensus.cc:2243] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.002516 1776 raft_consensus.cc:2272] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.003011 1774 ts_tablet_manager.cc:1916] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.003052 1776 ts_tablet_manager.cc:1916] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.006968 1767 tablet_bootstrap.cc:654] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.011966 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet a76d61c3464b41519d7657796bd853ff with delete_type TABLET_DATA_DELETED (replaced by tablet b240965605904533bd2af99f61aaef26) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.014868 1775 ts_tablet_manager.cc:1950] T 369cebfda6ee43638b79f8652fa55855 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:06.015988 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 342) took 3781 ms. Trace:
I20251024 19:48:06.016316 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet e3b4e85f741849b2a6a3e8574396ff88 with delete_type TABLET_DATA_DELETED (replaced by tablet 769f576810ab49368eb0fc7837b7da58) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:06.028883 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 593f5600e4274fce97e7bd33f025569f consensus state has no leader
I20251024 19:48:06.029309 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ebaca4c94c0a4718bce68ebbc4feb98a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:06.030671 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ee8861d72c804704af20dbe501b5332d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:06.043663 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f346a90a9efe442f82750336e3893deb" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:06.046402 1775 rpcz_store.cc:276] 1024 19:48:02.234599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.234683 (+ 84us) service_pool.cc:225] Handling call
1024 19:48:05.782121 (+3547438us) ts_tablet_manager.cc:1151] Deleting tablet 369cebfda6ee43638b79f8652fa55855
1024 19:48:05.782131 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:05.919949 (+137818us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.000654 (+ 80705us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.015931 (+ 15277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3547185}
I20251024 19:48:06.051136 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 369cebfda6ee43638b79f8652fa55855 (table default.Table_c3cbf88afafa464099c69d8e017743e8 [id=27d44c1a6a2046de8d4e634e4eb6eb7e]) successfully deleted
I20251024 19:48:06.053829 1775 tablet_replica.cc:333] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
W20251024 19:48:06.066787 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee8861d72c804704af20dbe501b5332d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.068243 1776 ts_tablet_manager.cc:1929] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251024 19:48:06.070052 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebaca4c94c0a4718bce68ebbc4feb98a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.031912 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet e20881e400cf4fac914b70afc79c084f (table default.Table_69e288a9d6d24a5f841f62cd9a208f0f [id=969d428d45db42d18a5a9b82ec945bfb]) successfully deleted
W20251024 19:48:06.073117 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 45cff9bcc4924e0081518c284f074940 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45cff9bcc4924e0081518c284f074940 already in progress: creating tablet
I20251024 19:48:06.073596 1776 log.cc:1199] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/857eed60ba2a4b309e86d88a0f093762
I20251024 19:48:06.074510 1776 ts_tablet_manager.cc:1950] T 857eed60ba2a4b309e86d88a0f093762 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:06.076905 1774 ts_tablet_manager.cc:1929] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.077082 1774 log.cc:1199] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/897db285f5c84930a5b438be1cb4eb96
I20251024 19:48:06.078418 2321 raft_consensus.cc:493] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:06.081540 2321 raft_consensus.cc:515] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.082293 2321 leader_election.cc:290] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:06.083757 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 419) took 2349 ms. Trace:
I20251024 19:48:06.086333 1776 rpcz_store.cc:276] 1024 19:48:03.733803 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.733862 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:06.002115 (+2268253us) ts_tablet_manager.cc:1151] Deleting tablet 857eed60ba2a4b309e86d88a0f093762
1024 19:48:06.002123 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.027753 (+ 25630us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.068073 (+ 40320us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.083693 (+ 15620us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2268047}
I20251024 19:48:06.087776 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 420) took 2310 ms. Trace:
I20251024 19:48:06.088198 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 857eed60ba2a4b309e86d88a0f093762 (table default.Table_b8e34cb629054ed9be6deaf4d30992a2 [id=d225099585ad4533b088821c639b5463]) successfully deleted
I20251024 19:48:06.089043 1776 rpcz_store.cc:276] 1024 19:48:03.777786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.778222 (+ 436us) service_pool.cc:225] Handling call
1024 19:48:06.087566 (+2309344us) ts_tablet_manager.cc:1151] Deleting tablet 2602335327fc489fa154056c5ef5138f
1024 19:48:06.087582 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.087714 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2308925}
I20251024 19:48:06.094689 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 434) took 1834 ms. Trace:
I20251024 19:48:06.096004 1776 rpcz_store.cc:276] 1024 19:48:04.259923 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.259994 (+ 71us) service_pool.cc:225] Handling call
1024 19:48:06.094481 (+1834487us) ts_tablet_manager.cc:1151] Deleting tablet d38154d3ffd34f4b8d7305b1c23a4fae
1024 19:48:06.094492 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.094618 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1834056}
W20251024 19:48:06.096436 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2602335327fc489fa154056c5ef5138f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2602335327fc489fa154056c5ef5138f already in progress: creating tablet
I20251024 19:48:06.097265 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 435) took 1832 ms. Trace:
W20251024 19:48:06.097642 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d38154d3ffd34f4b8d7305b1c23a4fae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d38154d3ffd34f4b8d7305b1c23a4fae already in progress: creating tablet
I20251024 19:48:06.098673 1776 rpcz_store.cc:276] 1024 19:48:04.264669 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.264739 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:06.097054 (+1832315us) ts_tablet_manager.cc:1151] Deleting tablet 4243f11bf2b34b30b10766c19e2c6409
1024 19:48:06.097069 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.097197 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1831901}
I20251024 19:48:06.099305 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 436) took 1831 ms. Trace:
I20251024 19:48:06.099434 1776 rpcz_store.cc:276] 1024 19:48:04.267937 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.269223 (+ 1286us) service_pool.cc:225] Handling call
1024 19:48:06.099080 (+1829857us) ts_tablet_manager.cc:1151] Deleting tablet d1d26e02ad2945b39cf27738ed5b7cf4
1024 19:48:06.099096 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.099237 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1829584}
I20251024 19:48:06.099928 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 437) took 1822 ms. Trace:
I20251024 19:48:06.100049 1776 rpcz_store.cc:276] 1024 19:48:04.277011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.277363 (+ 352us) service_pool.cc:225] Handling call
1024 19:48:06.099735 (+1822372us) ts_tablet_manager.cc:1151] Deleting tablet 4db17d9aff6542b6b94c09f872a4cceb
1024 19:48:06.099749 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.099865 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1822047}
I20251024 19:48:06.100529 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 438) took 1809 ms. Trace:
I20251024 19:48:06.100664 1776 rpcz_store.cc:276] 1024 19:48:04.291489 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.291557 (+ 68us) service_pool.cc:225] Handling call
1024 19:48:06.100326 (+1808769us) ts_tablet_manager.cc:1151] Deleting tablet ad81d80697984ff1834a7d795e0f4b9b
1024 19:48:06.100339 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.100471 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1808493}
I20251024 19:48:06.078804 1586 leader_election.cc:304] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:06.101351 2321 raft_consensus.cc:2749] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251024 19:48:06.102721 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ee03ede02c4bdab706fde910978332" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:06.102996 1508 raft_consensus.cc:2468] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:06.103307 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_581140b54cae4ac09f8e9c1b940e4241"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:06.104043 2321 consensus_queue.cc:1048] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:06.104874 1586 leader_election.cc:304] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:06.105165 2321 raft_consensus.cc:2804] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:06.105305 2321 raft_consensus.cc:493] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:06.105422 2321 raft_consensus.cc:3060] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:06.106248 1650 raft_consensus.cc:1275] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Refusing update from remote peer 10a9536207e343f188e0ef709081d04e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 19:48:06.107846 2020 raft_consensus.cc:3055] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Stepping down as leader of term 1
I20251024 19:48:06.108731 2020 raft_consensus.cc:740] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:06.108785 1775 raft_consensus.cc:2243] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.109035 1775 raft_consensus.cc:2272] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.108981 2321 raft_consensus.cc:515] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.109115 2020 consensus_queue.cc:260] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.109522 2020 raft_consensus.cc:3060] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Advancing to term 2
I20251024 19:48:06.109786 1775 ts_tablet_manager.cc:1916] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.110888 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a5cf2d40da674fc592fd529d58dc0575" } new_table_name: "default.Table_5c5192ae905d4d4d809ebaeff38c6ca8" modify_external_catalogs: true
I20251024 19:48:06.111325 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ee03ede02c4bdab706fde910978332" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:06.111577 1508 raft_consensus.cc:3060] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:06.115404 1508 raft_consensus.cc:2468] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:06.115904 2321 leader_election.cc:290] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:06.115983 1305 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 120243us
I20251024 19:48:06.116192 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b20ffadae0f24d4e996d915faeef9ddb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:06.116633 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4a69c0918d2e4eed9d91b1913ec2c21e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a69c0918d2e4eed9d91b1913ec2c21e already in progress: creating tablet
W20251024 19:48:06.117065 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2e7b3288182744969e217d4f174fe4ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e7b3288182744969e217d4f174fe4ea already in progress: creating tablet
W20251024 19:48:06.117998 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4243f11bf2b34b30b10766c19e2c6409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4243f11bf2b34b30b10766c19e2c6409 already in progress: creating tablet
I20251024 19:48:06.118149 1586 leader_election.cc:304] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:06.118279 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d1d26e02ad2945b39cf27738ed5b7cf4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1d26e02ad2945b39cf27738ed5b7cf4 already in progress: creating tablet
I20251024 19:48:06.118538 2321 raft_consensus.cc:2804] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
W20251024 19:48:06.118556 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4db17d9aff6542b6b94c09f872a4cceb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4db17d9aff6542b6b94c09f872a4cceb already in progress: creating tablet
I20251024 19:48:06.118482 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ee03ede02c4bdab706fde910978332" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:06.118773 2321 raft_consensus.cc:697] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:06.118798 1368 raft_consensus.cc:3060] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:06.119110 2321 consensus_queue.cc:237] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.119889 1367 raft_consensus.cc:1275] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Refusing update from remote peer 10a9536207e343f188e0ef709081d04e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 19:48:06.108355 2346 consensus_queue.cc:1048] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20251024 19:48:06.121498 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ee03ede02c4bdab706fde910978332" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:06.122975 1774 ts_tablet_manager.cc:1950] T 897db285f5c84930a5b438be1cb4eb96 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:06.123358 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 351) took 3135 ms. Trace:
I20251024 19:48:06.123610 1368 raft_consensus.cc:2468] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:06.123453 1774 rpcz_store.cc:276] 1024 19:48:02.987677 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.987738 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:06.001795 (+3014057us) ts_tablet_manager.cc:1151] Deleting tablet 897db285f5c84930a5b438be1cb4eb96
1024 19:48:06.001806 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.050021 (+ 48215us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.076758 (+ 26737us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.123297 (+ 46539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3013699}
I20251024 19:48:06.124593 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 356) took 3073 ms. Trace:
I20251024 19:48:06.124698 1774 rpcz_store.cc:276] 1024 19:48:03.050701 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.050755 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:06.124407 (+3073652us) ts_tablet_manager.cc:1151] Deleting tablet bc31e6c25ad14947af20abd766f12e78
1024 19:48:06.124430 (+ 23us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.124541 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3072961}
I20251024 19:48:06.125062 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 357) took 3073 ms. Trace:
I20251024 19:48:06.107563 2334 consensus_queue.cc:1048] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20251024 19:48:06.125157 1774 rpcz_store.cc:276] 1024 19:48:03.051253 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.051303 (+ 50us) service_pool.cc:225] Handling call
1024 19:48:06.124913 (+3073610us) ts_tablet_manager.cc:1151] Deleting tablet dc14b0cdb7f342ac956a7857ab3f35d7
1024 19:48:06.124923 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.125013 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3073393}
I20251024 19:48:06.117728 1775 ts_tablet_manager.cc:1929] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:06.125520 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 358) took 3073 ms. Trace:
I20251024 19:48:06.125648 1775 log.cc:1199] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/b9b1e0338fd14634a1e921735ae61289
I20251024 19:48:06.125636 1774 rpcz_store.cc:276] 1024 19:48:03.052047 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.052096 (+ 49us) service_pool.cc:225] Handling call
1024 19:48:06.125374 (+3073278us) ts_tablet_manager.cc:1151] Deleting tablet d76410afd7ba4c73836f4791ef739056
1024 19:48:06.125382 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.125460 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3073056}
I20251024 19:48:06.125943 1774 tablet_replica.cc:333] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:06.126204 1774 raft_consensus.cc:2243] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.126380 1775 ts_tablet_manager.cc:1950] T b9b1e0338fd14634a1e921735ae61289 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:06.126422 1774 raft_consensus.cc:2272] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.127080 1774 ts_tablet_manager.cc:1916] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.127610 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 343) took 3891 ms. Trace:
I20251024 19:48:06.128170 1775 rpcz_store.cc:276] 1024 19:48:02.235684 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.235752 (+ 68us) service_pool.cc:225] Handling call
1024 19:48:06.051489 (+3815737us) ts_tablet_manager.cc:1151] Deleting tablet b9b1e0338fd14634a1e921735ae61289
1024 19:48:06.051502 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.114004 (+ 62502us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.117557 (+ 3553us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.127546 (+ 9989us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54572,"tablet-delete.queue_time_us":3815404}
I20251024 19:48:06.129208 1775 tablet_replica.cc:333] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:06.130132 1775 raft_consensus.cc:2243] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.130908 1775 raft_consensus.cc:2272] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:06.131254 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59956dad06064024ab171631ef6c1dca consensus state has no leader
I20251024 19:48:06.133701 2272 consensus_queue.cc:1048] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:06.129293 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet b9b1e0338fd14634a1e921735ae61289 (table default.Table_f02338f14f2b44b3b89757b7edc59e32 [id=3e4514469cd64a8b86d9e44608c186ba]) successfully deleted
I20251024 19:48:06.140825 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 897db285f5c84930a5b438be1cb4eb96 (table default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]) successfully deleted
W20251024 19:48:06.141090 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet bc31e6c25ad14947af20abd766f12e78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc31e6c25ad14947af20abd766f12e78 already in progress: creating tablet
W20251024 19:48:06.141367 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet dc14b0cdb7f342ac956a7857ab3f35d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc14b0cdb7f342ac956a7857ab3f35d7 already in progress: creating tablet
W20251024 19:48:06.141695 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet d76410afd7ba4c73836f4791ef739056 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d76410afd7ba4c73836f4791ef739056 already in progress: creating tablet
W20251024 19:48:06.142578 1305 consensus_peers.cc:597] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 857eed60ba2a4b309e86d88a0f093762. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:06.145603 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b20ffadae0f24d4e996d915faeef9ddb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.148384 1767 tablet_bootstrap.cc:492] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:06.148623 1767 ts_tablet_manager.cc:1403] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.151s user 0.000s sys 0.017s
W20251024 19:48:06.149642 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_581140b54cae4ac09f8e9c1b940e4241 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.149824 1767 raft_consensus.cc:359] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.150120 1767 raft_consensus.cc:385] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.150228 1767 raft_consensus.cc:740] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.150557 1767 consensus_queue.cc:260] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.154691 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c7ecd114a8d44f5da18755c9a5ebed07" } new_table_name: "default.Table_5744ec49c5e349d79d7473d6ed4154c6" modify_external_catalogs: true
I20251024 19:48:06.156782 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 433) took 1897 ms. Trace:
I20251024 19:48:06.157871 1776 rpcz_store.cc:276] 1024 19:48:04.259779 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.362922 (+103143us) service_pool.cc:225] Handling call
1024 19:48:06.156542 (+1793620us) ts_tablet_manager.cc:1151] Deleting tablet af43ffe097e14c4faffed95e207d0893
1024 19:48:06.156556 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.156714 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1792377}
I20251024 19:48:06.159257 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 443) took 1494 ms. Trace:
W20251024 19:48:06.159832 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet af43ffe097e14c4faffed95e207d0893 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af43ffe097e14c4faffed95e207d0893 already in progress: creating tablet
I20251024 19:48:06.169790 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b82f8da2443f4f0f91dd2b762983abac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:06.170559 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b82f8da2443f4f0f91dd2b762983abac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.181025 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_31e350af7b2a4e38981c3db02a346e52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:06.182579 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31e350af7b2a4e38981c3db02a346e52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.186988 1776 rpcz_store.cc:276] 1024 19:48:04.664719 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.664784 (+ 65us) service_pool.cc:225] Handling call
1024 19:48:06.159027 (+1494243us) ts_tablet_manager.cc:1151] Deleting tablet 163ac050c8574615aef5061f563cfc0a
1024 19:48:06.159043 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.159180 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1494020}
W20251024 19:48:06.187570 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 163ac050c8574615aef5061f563cfc0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 163ac050c8574615aef5061f563cfc0a already in progress: creating tablet
I20251024 19:48:06.187729 1776 tablet_replica.cc:333] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:06.187721 2271 raft_consensus.cc:493] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:06.187891 2271 raft_consensus.cc:515] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:06.188798 2271 leader_election.cc:290] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:06.189272 1767 ts_tablet_manager.cc:1434] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.041s user 0.003s sys 0.000s
I20251024 19:48:06.189730 1767 tablet_bootstrap.cc:492] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:06.191905 1774 ts_tablet_manager.cc:1929] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.192699 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aebf76d4a66947ed9756643740418798" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:06.193388 1305 leader_election.cc:343] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.199261 1774 log.cc:1199] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/31647a48fb834c80b6920c756a94250d
I20251024 19:48:06.204466 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_03760ade20c7464f81dd428a3eb734fb" } new_table_name: "default.Table_1865562320ef412799f93a15c66ade0c" modify_external_catalogs: true
I20251024 19:48:06.206514 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f4e511aa3c4b46479c523daa4149707f" } new_table_name: "default.Table_f16963cab00043a4887d93a6326a1d9c" modify_external_catalogs: true
I20251024 19:48:06.136006 2173 consensus_queue.cc:1048] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:06.142228 2346 consensus_queue.cc:1048] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:06.145313 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 3754540ae21e4c65b5af3d0aa3f9189f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.154132 2224 consensus_queue.cc:1048] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:06.191382 2020 raft_consensus.cc:493] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251024 19:48:06.207859 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da7d38b63708427982591da1014aca81 consensus state has no leader
I20251024 19:48:06.207916 2020 raft_consensus.cc:515] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.208567 1775 ts_tablet_manager.cc:1916] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.208616 2020 leader_election.cc:290] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:06.192116 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aebf76d4a66947ed9756643740418798" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:06.208624 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_181f309333a24988954f97061ff12419"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:06.209152 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_181f309333a24988954f97061ff12419 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.210183 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd52295b14b14e70bbdcb45f1bd849e4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:06.210418 1648 raft_consensus.cc:2468] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:06.210803 1444 leader_election.cc:304] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:06.211112 2020 raft_consensus.cc:2804] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:06.211218 2020 raft_consensus.cc:493] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:06.211344 2020 raft_consensus.cc:3060] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:06.213480 1776 raft_consensus.cc:2243] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:06.193567 1766 tablet_bootstrap.cc:492] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.213763 1766 ts_tablet_manager.cc:1403] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.410s user 0.033s sys 0.044s
I20251024 19:48:06.214371 1776 raft_consensus.cc:2272] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.214324 2020 raft_consensus.cc:515] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.214839 1766 raft_consensus.cc:359] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.215098 2020 leader_election.cc:290] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:06.215617 1766 raft_consensus.cc:385] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.215816 1766 raft_consensus.cc:740] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.216056 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd52295b14b14e70bbdcb45f1bd849e4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:06.216248 1649 raft_consensus.cc:3060] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:06.217918 1766 consensus_queue.cc:260] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.218508 1766 ts_tablet_manager.cc:1434] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251024 19:48:06.219448 1649 raft_consensus.cc:2468] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
W20251024 19:48:06.220605 1308 leader_election.cc:343] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.220772 1308 leader_election.cc:304] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:06.224325 1768 tablet_bootstrap.cc:492] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:06.224522 1768 ts_tablet_manager.cc:1403] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.463s user 0.013s sys 0.018s
I20251024 19:48:06.225546 1768 raft_consensus.cc:359] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.225773 1768 raft_consensus.cc:385] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.225893 1768 raft_consensus.cc:740] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.226363 1776 ts_tablet_manager.cc:1916] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.226742 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd52295b14b14e70bbdcb45f1bd849e4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:06.226990 1372 raft_consensus.cc:2468] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:06.227463 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd52295b14b14e70bbdcb45f1bd849e4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:06.227672 1372 raft_consensus.cc:3060] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:06.235312 1767 tablet_bootstrap.cc:654] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.236434 1774 ts_tablet_manager.cc:1950] T 31647a48fb834c80b6920c756a94250d P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:06.242791 1766 tablet_bootstrap.cc:492] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.244930 1766 tablet_bootstrap.cc:654] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.245642 1444 leader_election.cc:304] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:06.246125 2334 raft_consensus.cc:2804] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:06.246500 2334 raft_consensus.cc:697] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:06.247181 2334 consensus_queue.cc:237] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:06.253844 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b22b58404b8b441ba97ef536885a66e0 consensus state has no leader
I20251024 19:48:06.258805 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d18d308917fc44cbb7f0af85912d01c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:06.259218 2369 tablet.cc:1722] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:06.259613 1776 ts_tablet_manager.cc:1929] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.259760 1776 log.cc:1199] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/7c357a2a81064b3891afda6d252898b9
W20251024 19:48:06.259765 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d18d308917fc44cbb7f0af85912d01c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.262634 2367 tablet.cc:1722] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:06.272634 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a12e415590a4c71aab4ad7d30412c83 with delete_type TABLET_DATA_DELETED (replaced by tablet d6de2c21796f4bc4904f88b654c3baec) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.226140 1768 consensus_queue.cc:260] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.273912 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3819751f36bf419e9edd0f1ff339713a" } new_table_name: "default.Table_ea0d3589c13a4ebeb950ddc737a9948b" modify_external_catalogs: true
I20251024 19:48:06.274093 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 361) took 3038 ms. Trace:
I20251024 19:48:06.279836 2370 tablet.cc:1722] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251024 19:48:06.281649 1447 consensus_peers.cc:597] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31647a48fb834c80b6920c756a94250d. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:06.285732 2366 tablet.cc:1722] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:06.285985 1188 catalog_manager.cc:5649] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:06.288579 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 46c1e2cef6cf4772b624060afb6fb117 (DEFAULT_TABLE table=default.Table_86c533ff0d26405bb7a1c203c39d18a0 [id=9f790b431caf410db28053df4923702a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.289088 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c1e2cef6cf4772b624060afb6fb117. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.289050 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 46c1e2cef6cf4772b624060afb6fb117 (DEFAULT_TABLE table=default.Table_86c533ff0d26405bb7a1c203c39d18a0 [id=9f790b431caf410db28053df4923702a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.289485 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c1e2cef6cf4772b624060afb6fb117. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.291361 1767 tablet_bootstrap.cc:492] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:06.291522 1767 ts_tablet_manager.cc:1403] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.102s user 0.005s sys 0.005s
I20251024 19:48:06.292296 1767 raft_consensus.cc:359] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.292481 1767 raft_consensus.cc:385] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.291994 1774 rpcz_store.cc:276] 1024 19:48:03.216439 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.216500 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:06.125853 (+2909353us) ts_tablet_manager.cc:1151] Deleting tablet 31647a48fb834c80b6920c756a94250d
1024 19:48:06.125861 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.163800 (+ 37939us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.191756 (+ 27956us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.254404 (+ 62648us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4864,"tablet-delete.queue_time_us":2909117}
I20251024 19:48:06.292573 1767 raft_consensus.cc:740] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.292865 1767 consensus_queue.cc:260] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.293634 1767 ts_tablet_manager.cc:1434] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:06.295006 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 31647a48fb834c80b6920c756a94250d (table default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]) successfully deleted
I20251024 19:48:06.295056 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 46c1e2cef6cf4772b624060afb6fb117 (DEFAULT_TABLE table=default.Table_86c533ff0d26405bb7a1c203c39d18a0 [id=9f790b431caf410db28053df4923702a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.295493 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c1e2cef6cf4772b624060afb6fb117. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.299547 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_407e3b1465004f64bcabe7e2da5d4935"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:06.299829 1767 tablet_bootstrap.cc:492] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:48:06.300418 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_407e3b1465004f64bcabe7e2da5d4935 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.304590 1775 ts_tablet_manager.cc:1929] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:06.304804 1775 log.cc:1199] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/90405a0c13bd4c229b011c90232f751d
I20251024 19:48:06.305258 1775 ts_tablet_manager.cc:1950] T 90405a0c13bd4c229b011c90232f751d P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:06.288579 1776 ts_tablet_manager.cc:1950] T 7c357a2a81064b3891afda6d252898b9 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:06.311249 2271 raft_consensus.cc:2749] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:06.311779 1774 tablet_replica.cc:333] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:06.312862 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_accb35a3be634901b43dbbdf579098cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:06.313027 1624 tablet_service.cc:1505] Processing CreateTablet for tablet f20392eb64f64f96a8121b3641d0ee30 (DEFAULT_TABLE table=default.Table_dbcb9a24f3f34a91b79bbc952d4f0012 [id=2f233048481949b6a9c5d93dc8f58740]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.314332 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f20392eb64f64f96a8121b3641d0ee30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.315861 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 445) took 1632 ms. Trace:
I20251024 19:48:06.321872 1620 tablet_service.cc:1505] Processing CreateTablet for tablet af420ac6b4704b6fa41b4719f5bf9d3d (DEFAULT_TABLE table=default.Table_783450f8b33e4b2b936136effef29fc6 [id=c73315d3404c4478bcce1ecf143e46ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.313074 1488 tablet_service.cc:1505] Processing CreateTablet for tablet af420ac6b4704b6fa41b4719f5bf9d3d (DEFAULT_TABLE table=default.Table_783450f8b33e4b2b936136effef29fc6 [id=c73315d3404c4478bcce1ecf143e46ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.317350 2334 consensus_queue.cc:1048] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:06.322826 1444 consensus_peers.cc:597] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e20881e400cf4fac914b70afc79c084f. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:06.324087 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af420ac6b4704b6fa41b4719f5bf9d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.330380 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 344) took 4071 ms. Trace:
I20251024 19:48:06.333323 1776 rpcz_store.cc:276] 1024 19:48:04.678824 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.678889 (+ 65us) service_pool.cc:225] Handling call
1024 19:48:06.187599 (+1508710us) ts_tablet_manager.cc:1151] Deleting tablet 7c357a2a81064b3891afda6d252898b9
1024 19:48:06.187613 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.214003 (+ 26390us) spinlock_profiling.cc:243] Waited 25.5 ms on lock 0x616000036154. stack: 00007fea3c4169f6 00007fea3c4167b0 00007fea3b3f23ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fea482dc100 00007fea49629ae3 00007fea4c1e6287 00007fea4c2192d4 00007fea4c1f9d70 00007fea4c1f9afd 0000000000a8867c 00007fea3c45cd7f 00007fea3c46004b 00007fea3c45fec8
1024 19:48:06.254817 (+ 40814us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.259472 (+ 4655us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.311397 (+ 51925us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":58705280,"tablet-delete.queue_time_us":1508482}
I20251024 19:48:06.333747 1776 tablet_replica.cc:333] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:06.341456 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af420ac6b4704b6fa41b4719f5bf9d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.341677 1767 tablet_bootstrap.cc:654] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.341756 1776 raft_consensus.cc:2243] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.347193 1776 raft_consensus.cc:2272] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.348052 1776 ts_tablet_manager.cc:1916] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:06.348532 1584 connection.cc:537] client connection to 127.0.252.254:36505 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251024 19:48:06.348613 1306 connection.cc:537] client connection to 127.0.252.254:36505 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251024 19:48:06.334138 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 7c357a2a81064b3891afda6d252898b9 (table default.Table_c570270b7d7b4f5091c34e0550dd46ef [id=ed879934f5754b92b8cff234f3f66333]) successfully deleted
I20251024 19:48:06.344955 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 780442b310e6409498e6bbe4256280a4
I20251024 19:48:06.345211 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 65424b00ab9c454f9584048caba83e6a
W20251024 19:48:06.363353 1177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.252.254:36505 (error 108) [suppressed 8 similar messages]
I20251024 19:48:06.345422 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3324755a5a184a609fae5f7a25d58e4c
I20251024 19:48:06.345634 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7eb62791100f477784ab315923b6ceb6
I20251024 19:48:06.369761 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 780442b310e6409498e6bbe4256280a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:06.371073 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3259823b622461fa823841925b3d203 consensus state has no leader
I20251024 19:48:06.372083 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 3324755a5a184a609fae5f7a25d58e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:06.374292 1351 tablet_service.cc:1505] Processing CreateTablet for tablet f20392eb64f64f96a8121b3641d0ee30 (DEFAULT_TABLE table=default.Table_dbcb9a24f3f34a91b79bbc952d4f0012 [id=2f233048481949b6a9c5d93dc8f58740]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.374775 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f20392eb64f64f96a8121b3641d0ee30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.376171 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 65424b00ab9c454f9584048caba83e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:06.376351 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet 7eb62791100f477784ab315923b6ceb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:06.376636 2379 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 19:48:06.349540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.252.254:36505 (local address 127.0.252.195:34237)
1024 19:48:06.376167 (+ 26627us) negotiation.cc:107] Waiting for socket to connect
1024 19:48:06.376357 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.252.254:36505: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":26498,"threads_started":1}
I20251024 19:48:06.377417 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 780442b310e6409498e6bbe4256280a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:06.377784 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_accb35a3be634901b43dbbdf579098cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.378054 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 65424b00ab9c454f9584048caba83e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.378103 1344 tablet_service.cc:1505] Processing CreateTablet for tablet af420ac6b4704b6fa41b4719f5bf9d3d (DEFAULT_TABLE table=default.Table_783450f8b33e4b2b936136effef29fc6 [id=c73315d3404c4478bcce1ecf143e46ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.378536 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 3324755a5a184a609fae5f7a25d58e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.379133 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 7eb62791100f477784ab315923b6ceb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.380508 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 55986287a8bb40c8bb28e536fb22fb27 (DEFAULT_TABLE table=default.Table_86b9f6987402431196ad8064631c34bc [id=8dd1c89095be4b70abe16d76832986cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.323567 1766 tablet_bootstrap.cc:492] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.381798 1766 ts_tablet_manager.cc:1403] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.163s user 0.001s sys 0.005s
I20251024 19:48:06.381687 1775 rpcz_store.cc:276] 1024 19:48:02.258909 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.258992 (+ 83us) service_pool.cc:225] Handling call
1024 19:48:06.129100 (+3870108us) ts_tablet_manager.cc:1151] Deleting tablet 90405a0c13bd4c229b011c90232f751d
1024 19:48:06.129112 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.212264 (+ 83152us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.304437 (+ 92173us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.330340 (+ 25903us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3869781}
I20251024 19:48:06.382165 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 90405a0c13bd4c229b011c90232f751d (table default.Table_eb9c8019451d46fba31a87a0b6ac6ad8 [id=e5b1dd31c2114e8a95ba33981029255c]) successfully deleted
W20251024 19:48:06.348259 1445 connection.cc:537] client connection to 127.0.252.254:36505 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251024 19:48:06.382707 1766 raft_consensus.cc:359] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.382946 1766 raft_consensus.cc:385] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.383044 1766 raft_consensus.cc:740] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.383117 1767 tablet_bootstrap.cc:492] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:06.367683 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 3324755a5a184a609fae5f7a25d58e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.383266 1767 ts_tablet_manager.cc:1403] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.089s user 0.001s sys 0.005s
I20251024 19:48:06.383270 1766 consensus_queue.cc:260] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.383808 1766 ts_tablet_manager.cc:1434] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:06.367805 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 780442b310e6409498e6bbe4256280a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.368008 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 65424b00ab9c454f9584048caba83e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.384050 1767 raft_consensus.cc:359] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.368145 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 7eb62791100f477784ab315923b6ceb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.376603 1476 tablet_service.cc:1505] Processing CreateTablet for tablet f20392eb64f64f96a8121b3641d0ee30 (DEFAULT_TABLE table=default.Table_dbcb9a24f3f34a91b79bbc952d4f0012 [id=2f233048481949b6a9c5d93dc8f58740]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.384264 1767 raft_consensus.cc:385] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.384342 1767 raft_consensus.cc:740] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.384584 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f20392eb64f64f96a8121b3641d0ee30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.384104 1766 tablet_bootstrap.cc:492] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.384588 1767 consensus_queue.cc:260] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.385298 1767 ts_tablet_manager.cc:1434] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:06.385663 1767 tablet_bootstrap.cc:492] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:06.386567 1766 tablet_bootstrap.cc:654] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.387305 1767 tablet_bootstrap.cc:654] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.388978 1372 raft_consensus.cc:2468] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:06.390245 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 354) took 3891 ms. Trace:
I20251024 19:48:06.390458 1775 rpcz_store.cc:276] 1024 19:48:02.498692 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.499122 (+ 430us) service_pool.cc:225] Handling call
1024 19:48:06.390005 (+3890883us) ts_tablet_manager.cc:1151] Deleting tablet 5d129bdc876e425aabcf1c99997e319a
1024 19:48:06.390019 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.390197 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3886503}
W20251024 19:48:06.391119 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 5d129bdc876e425aabcf1c99997e319a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d129bdc876e425aabcf1c99997e319a already in progress: creating tablet
I20251024 19:48:06.391281 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af420ac6b4704b6fa41b4719f5bf9d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.391307 2020 consensus_queue.cc:1048] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:06.391625 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 356) took 3892 ms. Trace:
I20251024 19:48:06.391851 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55986287a8bb40c8bb28e536fb22fb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.391739 1775 rpcz_store.cc:276] 1024 19:48:02.499034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.499122 (+ 88us) service_pool.cc:225] Handling call
1024 19:48:06.391483 (+3892361us) ts_tablet_manager.cc:1151] Deleting tablet 129debb55f4a4b7295a8ad88e55ebc83
1024 19:48:06.391492 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.391584 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3883404}
I20251024 19:48:06.392138 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 355) took 3893 ms. Trace:
W20251024 19:48:06.392182 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 129debb55f4a4b7295a8ad88e55ebc83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 129debb55f4a4b7295a8ad88e55ebc83 already in progress: creating tablet
I20251024 19:48:06.392262 1775 rpcz_store.cc:276] 1024 19:48:02.498865 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.499150 (+ 285us) service_pool.cc:225] Handling call
1024 19:48:06.392018 (+3892868us) ts_tablet_manager.cc:1151] Deleting tablet f3d80885fcc642b8a75da30cd4abecbd
1024 19:48:06.392026 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.392109 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3883089}
I20251024 19:48:06.393096 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 360) took 3880 ms. Trace:
I20251024 19:48:06.393230 1775 rpcz_store.cc:276] 1024 19:48:02.515566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.515630 (+ 64us) service_pool.cc:225] Handling call
1024 19:48:06.392919 (+3877289us) ts_tablet_manager.cc:1151] Deleting tablet 607ae453221d422e9d07882830c02275
1024 19:48:06.392949 (+ 30us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.393046 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3874919}
I20251024 19:48:06.394454 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 55986287a8bb40c8bb28e536fb22fb27 (DEFAULT_TABLE table=default.Table_86b9f6987402431196ad8064631c34bc [id=8dd1c89095be4b70abe16d76832986cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.394984 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55986287a8bb40c8bb28e536fb22fb27. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:06.396282 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 607ae453221d422e9d07882830c02275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607ae453221d422e9d07882830c02275 already in progress: creating tablet
I20251024 19:48:06.400236 1766 tablet_bootstrap.cc:492] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.400422 1766 ts_tablet_manager.cc:1403] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.008s
I20251024 19:48:06.401172 1766 raft_consensus.cc:359] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.401396 1766 raft_consensus.cc:385] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.401473 1766 raft_consensus.cc:740] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.401726 1766 consensus_queue.cc:260] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.402400 1766 ts_tablet_manager.cc:1434] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251024 19:48:06.402715 1766 tablet_bootstrap.cc:492] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.402741 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 55986287a8bb40c8bb28e536fb22fb27 (DEFAULT_TABLE table=default.Table_86b9f6987402431196ad8064631c34bc [id=8dd1c89095be4b70abe16d76832986cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.403245 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55986287a8bb40c8bb28e536fb22fb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.404984 1766 tablet_bootstrap.cc:654] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.414839 1766 tablet_bootstrap.cc:492] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.415002 1766 ts_tablet_manager.cc:1403] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20251024 19:48:06.415839 1766 raft_consensus.cc:359] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.416059 1766 raft_consensus.cc:385] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.416162 1766 raft_consensus.cc:740] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.418396 1766 consensus_queue.cc:260] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.419155 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 361) took 3899 ms. Trace:
I20251024 19:48:06.419325 1775 rpcz_store.cc:276] 1024 19:48:02.519514 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.519710 (+ 196us) service_pool.cc:225] Handling call
1024 19:48:06.418962 (+3899252us) ts_tablet_manager.cc:1151] Deleting tablet d9bf08ccf7054fedb9efcbb00354c71a
1024 19:48:06.418973 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.419115 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3897431}
I20251024 19:48:06.420288 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 363) took 3893 ms. Trace:
W20251024 19:48:06.421409 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet d9bf08ccf7054fedb9efcbb00354c71a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9bf08ccf7054fedb9efcbb00354c71a already in progress: creating tablet
I20251024 19:48:06.422354 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3114365d011943a389d8d35215265bbf" } new_table_name: "default.Table_5114b358c69441539cfc49077ea63f9e" modify_external_catalogs: true
I20251024 19:48:06.423552 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2fb3f9af398745dcafa14f615ef6a116"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:06.427932 1624 tablet_service.cc:1505] Processing CreateTablet for tablet d4b3a50db1714eb38cc0ea03f612fc95 (DEFAULT_TABLE table=default.Table_5fd79a379e3b43aeb80de8af15bf4427 [id=3b8593f5874f43f9af9da22233984dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.428829 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4b3a50db1714eb38cc0ea03f612fc95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.431236 1775 rpcz_store.cc:276] 1024 19:48:02.526508 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.526785 (+ 277us) service_pool.cc:225] Handling call
1024 19:48:06.420154 (+3893369us) ts_tablet_manager.cc:1151] Deleting tablet 80c58e2439dc490c824b5f0023b5434b
1024 19:48:06.420165 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.420248 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3892932}
I20251024 19:48:06.432032 1766 ts_tablet_manager.cc:1434] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251024 19:48:06.432781 1766 tablet_bootstrap.cc:492] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.433416 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 94b35252e63f4a8ba305cc029ddb5ae8
I20251024 19:48:06.433157 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 372) took 3653 ms. Trace:
I20251024 19:48:06.433876 1775 rpcz_store.cc:276] 1024 19:48:02.779926 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.780375 (+ 449us) service_pool.cc:225] Handling call
1024 19:48:06.433013 (+3652638us) ts_tablet_manager.cc:1151] Deleting tablet b79801594e95434cbb8fe7b75b14602e
1024 19:48:06.433022 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.433126 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3652294}
W20251024 19:48:06.435350 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 80c58e2439dc490c824b5f0023b5434b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80c58e2439dc490c824b5f0023b5434b already in progress: creating tablet
I20251024 19:48:06.435527 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 83b91a5d764449d5b82ef6d1ee0360f5
I20251024 19:48:06.435674 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_05b0fc0694f240dda1d013cc7e547b82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:06.438155 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fb3f9af398745dcafa14f615ef6a116 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.437987 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 94b35252e63f4a8ba305cc029ddb5ae8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b97907234e543c2806f40619046f6ae) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.438844 1775 tablet_replica.cc:333] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:06.439044 1775 raft_consensus.cc:2243] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.439203 1775 raft_consensus.cc:2272] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.439335 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 83b91a5d764449d5b82ef6d1ee0360f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.439558 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet 94b35252e63f4a8ba305cc029ddb5ae8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b97907234e543c2806f40619046f6ae) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.440249 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_370e897b37284a38948769e22c2d22f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:06.440537 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 83b91a5d764449d5b82ef6d1ee0360f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.440929 1776 ts_tablet_manager.cc:1929] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.441067 1776 log.cc:1199] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/2496f103436047dea5dabb577651249f
I20251024 19:48:06.441557 1776 ts_tablet_manager.cc:1950] T 2496f103436047dea5dabb577651249f P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:06.442378 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 446) took 1763 ms. Trace:
I20251024 19:48:06.442493 1776 rpcz_store.cc:276] 1024 19:48:04.679273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.679473 (+ 200us) service_pool.cc:225] Handling call
1024 19:48:06.333678 (+1654205us) ts_tablet_manager.cc:1151] Deleting tablet 2496f103436047dea5dabb577651249f
1024 19:48:06.333687 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.404339 (+ 70652us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.440811 (+ 36472us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.442317 (+ 1506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654002}
W20251024 19:48:06.443305 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_370e897b37284a38948769e22c2d22f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.444376 1192 catalog_manager.cc:6719] ReplaceTablet: tablet 94b35252e63f4a8ba305cc029ddb5ae8 deleted and replaced by tablet 4b97907234e543c2806f40619046f6ae
I20251024 19:48:06.445237 1775 ts_tablet_manager.cc:1916] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:06.445757 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05b0fc0694f240dda1d013cc7e547b82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.445950 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 2496f103436047dea5dabb577651249f (table default.Table_9ecacbf3e0fe41b7b54b20dbd81c4e05 [id=4e52676da82e4f5f807ffc32fb747281]) successfully deleted
I20251024 19:48:06.446457 1767 tablet_bootstrap.cc:492] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:06.446676 1767 ts_tablet_manager.cc:1403] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.000s
I20251024 19:48:06.447096 1768 ts_tablet_manager.cc:1434] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.222s user 0.000s sys 0.012s
I20251024 19:48:06.447463 1768 tablet_bootstrap.cc:492] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:06.447674 1185 catalog_manager.cc:5649] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } }
I20251024 19:48:06.447752 1767 raft_consensus.cc:359] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.448010 1767 raft_consensus.cc:385] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.448133 1767 raft_consensus.cc:740] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.448818 1774 raft_consensus.cc:2243] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:06.449003 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 1d04a9b54bd6446d8deb64292cef672d (table default.Table_147e888401aa40e5a60d97884b36cb98 [id=1acdc5d0bfb94e0faf47055b20d803b0]). Expected version 1 got 0
I20251024 19:48:06.449182 1774 raft_consensus.cc:2272] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.449252 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]). Expected version 1 got 0
I20251024 19:48:06.449890 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet ad055dc3387b414bb55d09b9683ba17e (table default.Table_cde3670bf0d04dc0b2049193bcee10d4 [id=c3de6937e93a48dcb88cabee59a09d32]). Expected version 1 got 0
I20251024 19:48:06.449955 1774 ts_tablet_manager.cc:1916] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.451443 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3c5c40909920463ab35ba50c7a03bd36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:06.451956 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c5c40909920463ab35ba50c7a03bd36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.452026 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 94b35252e63f4a8ba305cc029ddb5ae8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b97907234e543c2806f40619046f6ae) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:06.452260 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 83b91a5d764449d5b82ef6d1ee0360f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:06.454152 2337 raft_consensus.cc:493] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:06.454295 2337 raft_consensus.cc:515] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.455256 2337 leader_election.cc:290] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:06.455890 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a12e415590a4c71aab4ad7d30412c83" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:06.456387 1305 leader_election.cc:343] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.461036 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4f873edb1c0d444a88c3485b0aaa7623"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:06.449697 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_25c67815f6fc45c780e1a9fd8ae678f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:06.462325 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f873edb1c0d444a88c3485b0aaa7623 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:06.463886 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25c67815f6fc45c780e1a9fd8ae678f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.464802 1336 tablet_service.cc:1505] Processing CreateTablet for tablet d4b3a50db1714eb38cc0ea03f612fc95 (DEFAULT_TABLE table=default.Table_5fd79a379e3b43aeb80de8af15bf4427 [id=3b8593f5874f43f9af9da22233984dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.465127 1766 tablet_bootstrap.cc:654] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:06.467078 1177 consensus_peers.cc:597] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c -> Peer ef8906492b1a4368936fac95c11e1f73 (127.0.252.254:36505): Couldn't send request to peer ef8906492b1a4368936fac95c11e1f73. Status: Network error: Client connection negotiation failed: client connection to 127.0.252.254:36505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:06.467752 1488 tablet_service.cc:1505] Processing CreateTablet for tablet d4b3a50db1714eb38cc0ea03f612fc95 (DEFAULT_TABLE table=default.Table_5fd79a379e3b43aeb80de8af15bf4427 [id=3b8593f5874f43f9af9da22233984dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.468408 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4b3a50db1714eb38cc0ea03f612fc95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.471873 1624 tablet_service.cc:1505] Processing CreateTablet for tablet ca95e470a3db47ceb2da9246e4bc094a (DEFAULT_TABLE table=default.Table_f052d9f6feec4b35be9654644bbb59c9 [id=d3271af922bc4446a73cb679761e3bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.472290 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca95e470a3db47ceb2da9246e4bc094a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.475315 1474 tablet_service.cc:1505] Processing CreateTablet for tablet ca95e470a3db47ceb2da9246e4bc094a (DEFAULT_TABLE table=default.Table_f052d9f6feec4b35be9654644bbb59c9 [id=d3271af922bc4446a73cb679761e3bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.475685 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca95e470a3db47ceb2da9246e4bc094a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.476691 1776 tablet_replica.cc:333] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:06.476919 1776 raft_consensus.cc:2243] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:06.478902 1776 pending_rounds.cc:70] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:06.478992 1776 pending_rounds.cc:77] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429329220100096 op_type: NO_OP noop_request { }
I20251024 19:48:06.479132 1776 raft_consensus.cc:2889] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:06.479256 1776 raft_consensus.cc:2272] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:06.481240 1308 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 271240us
I20251024 19:48:06.481742 1767 consensus_queue.cc:260] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.483846 1766 tablet_bootstrap.cc:492] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.484032 1766 ts_tablet_manager.cc:1403] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.001s
I20251024 19:48:06.484771 1768 tablet_bootstrap.cc:654] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.484818 1766 raft_consensus.cc:359] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:06.485028 1766 raft_consensus.cc:385] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.485127 1766 raft_consensus.cc:740] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.485369 1766 consensus_queue.cc:260] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:06.485467 1776 ts_tablet_manager.cc:1916] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.483274 1767 ts_tablet_manager.cc:1434] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.036s user 0.002s sys 0.001s
I20251024 19:48:06.488713 1767 tablet_bootstrap.cc:492] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:06.490782 1767 tablet_bootstrap.cc:654] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.500288 1775 ts_tablet_manager.cc:1929] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.501497 1775 log.cc:1199] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/9623d7baf0174a019e2353a5effef081
I20251024 19:48:06.502887 1776 ts_tablet_manager.cc:1929] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.503489 1775 ts_tablet_manager.cc:1950] T 9623d7baf0174a019e2353a5effef081 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:06.504246 1766 ts_tablet_manager.cc:1434] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20251024 19:48:06.504882 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 373) took 3723 ms. Trace:
I20251024 19:48:06.505169 1776 log.cc:1199] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/346383e85cc64b3a8da70a71868ceccc
I20251024 19:48:06.505360 1775 rpcz_store.cc:276] 1024 19:48:02.780863 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.781054 (+ 191us) service_pool.cc:225] Handling call
1024 19:48:06.438785 (+3657731us) ts_tablet_manager.cc:1151] Deleting tablet 9623d7baf0174a019e2353a5effef081
1024 19:48:06.438793 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.453065 (+ 14272us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.500173 (+ 47108us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.504292 (+ 4119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3657554}
I20251024 19:48:06.505564 1766 tablet_bootstrap.cc:492] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.506480 1776 ts_tablet_manager.cc:1950] T 346383e85cc64b3a8da70a71868ceccc P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:06.507807 1766 tablet_bootstrap.cc:654] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.508080 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 447) took 1828 ms. Trace:
I20251024 19:48:06.508301 1776 rpcz_store.cc:276] 1024 19:48:04.679402 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.679721 (+ 319us) service_pool.cc:225] Handling call
1024 19:48:06.476587 (+1796866us) ts_tablet_manager.cc:1151] Deleting tablet 346383e85cc64b3a8da70a71868ceccc
1024 19:48:06.476596 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.495107 (+ 18511us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.502787 (+ 7680us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.507956 (+ 5169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1796694}
I20251024 19:48:06.509191 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 448) took 1829 ms. Trace:
I20251024 19:48:06.509400 1776 rpcz_store.cc:276] 1024 19:48:04.680067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.680109 (+ 42us) service_pool.cc:225] Handling call
1024 19:48:06.508877 (+1828768us) ts_tablet_manager.cc:1151] Deleting tablet f83b9290f00145a780318a43249eec12
1024 19:48:06.508886 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.509102 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1828585}
I20251024 19:48:06.510222 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 449) took 1781 ms. Trace:
I20251024 19:48:06.510447 1776 rpcz_store.cc:276] 1024 19:48:04.728483 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.728570 (+ 87us) service_pool.cc:225] Handling call
1024 19:48:06.509890 (+1781320us) ts_tablet_manager.cc:1151] Deleting tablet 9b89a90bb4ca447fbaf34fe39001c261
1024 19:48:06.509928 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.510140 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1781049}
I20251024 19:48:06.511225 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 453) took 1733 ms. Trace:
I20251024 19:48:06.511871 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 346383e85cc64b3a8da70a71868ceccc (table default.Table_acb41c25b40742ef92f4bb9d45ada0a3 [id=44c9c172ae1242d3a7ac272857209ff8]) successfully deleted
W20251024 19:48:06.512205 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f83b9290f00145a780318a43249eec12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f83b9290f00145a780318a43249eec12 already in progress: creating tablet
I20251024 19:48:06.511688 1776 rpcz_store.cc:276] 1024 19:48:04.777467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.777531 (+ 64us) service_pool.cc:225] Handling call
1024 19:48:06.510935 (+1733404us) ts_tablet_manager.cc:1151] Deleting tablet 3cf4596b333244cc9e9e35a128721e4a
1024 19:48:06.510948 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.511126 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1733074}
W20251024 19:48:06.512463 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9b89a90bb4ca447fbaf34fe39001c261 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b89a90bb4ca447fbaf34fe39001c261 already in progress: creating tablet
W20251024 19:48:06.512686 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3cf4596b333244cc9e9e35a128721e4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cf4596b333244cc9e9e35a128721e4a already in progress: creating tablet
I20251024 19:48:06.512778 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 457) took 1693 ms. Trace:
I20251024 19:48:06.512988 1776 rpcz_store.cc:276] 1024 19:48:04.819044 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.819095 (+ 51us) service_pool.cc:225] Handling call
1024 19:48:06.512489 (+1693394us) ts_tablet_manager.cc:1151] Deleting tablet ebcc09d8e029451d8fb6b6601ad16087
1024 19:48:06.512503 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.512658 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693172}
I20251024 19:48:06.513684 1776 tablet_replica.cc:333] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:06.514173 1776 raft_consensus.cc:2243] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:06.514832 1776 raft_consensus.cc:2272] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:06.515781 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet ebcc09d8e029451d8fb6b6601ad16087 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebcc09d8e029451d8fb6b6601ad16087 already in progress: creating tablet
I20251024 19:48:06.515863 1776 ts_tablet_manager.cc:1916] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.515821 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 381) took 3727 ms. Trace:
I20251024 19:48:06.516134 1775 rpcz_store.cc:276] 1024 19:48:02.788768 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.866714 (+ 77946us) service_pool.cc:225] Handling call
1024 19:48:06.515508 (+3648794us) ts_tablet_manager.cc:1151] Deleting tablet 4a69c0918d2e4eed9d91b1913ec2c21e
1024 19:48:06.515545 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.515732 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3626825}
I20251024 19:48:06.516588 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 9623d7baf0174a019e2353a5effef081 (table default.Table_7f14fdc529ec48d68d03ca041f64a53f [id=281060b712614bbeb46a006b6e34a91e]) successfully deleted
W20251024 19:48:06.516918 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4a69c0918d2e4eed9d91b1913ec2c21e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a69c0918d2e4eed9d91b1913ec2c21e already in progress: creating tablet
I20251024 19:48:06.517220 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 382) took 3728 ms. Trace:
I20251024 19:48:06.517437 1775 rpcz_store.cc:276] 1024 19:48:02.788867 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:02.878282 (+ 89415us) service_pool.cc:225] Handling call
1024 19:48:06.516881 (+3638599us) ts_tablet_manager.cc:1151] Deleting tablet 2e7b3288182744969e217d4f174fe4ea
1024 19:48:06.516891 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.517114 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3603291}
W20251024 19:48:06.518548 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2e7b3288182744969e217d4f174fe4ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e7b3288182744969e217d4f174fe4ea already in progress: creating tablet
I20251024 19:48:06.519687 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 389) took 3373 ms. Trace:
I20251024 19:48:06.520159 1775 rpcz_store.cc:276] 1024 19:48:03.146615 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.146740 (+ 125us) service_pool.cc:225] Handling call
1024 19:48:06.519104 (+3372364us) ts_tablet_manager.cc:1151] Deleting tablet bc31e6c25ad14947af20abd766f12e78
1024 19:48:06.519205 (+ 101us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.519593 (+ 388us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3372098}
W20251024 19:48:06.520890 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet bc31e6c25ad14947af20abd766f12e78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc31e6c25ad14947af20abd766f12e78 already in progress: creating tablet
I20251024 19:48:06.521605 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 390) took 3374 ms. Trace:
I20251024 19:48:06.521844 1775 rpcz_store.cc:276] 1024 19:48:03.146744 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.147032 (+ 288us) service_pool.cc:225] Handling call
1024 19:48:06.521331 (+3374299us) ts_tablet_manager.cc:1151] Deleting tablet dc14b0cdb7f342ac956a7857ab3f35d7
1024 19:48:06.521380 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.521509 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3374135}
I20251024 19:48:06.522487 1766 tablet_bootstrap.cc:492] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.522670 1766 ts_tablet_manager.cc:1403] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
W20251024 19:48:06.522900 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet dc14b0cdb7f342ac956a7857ab3f35d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc14b0cdb7f342ac956a7857ab3f35d7 already in progress: creating tablet
I20251024 19:48:06.523488 1766 raft_consensus.cc:359] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.523681 1766 raft_consensus.cc:385] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.523789 1766 raft_consensus.cc:740] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.524031 1766 consensus_queue.cc:260] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.524147 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 391) took 3377 ms. Trace:
I20251024 19:48:06.525458 1775 rpcz_store.cc:276] 1024 19:48:03.146869 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.147224 (+ 355us) service_pool.cc:225] Handling call
1024 19:48:06.523307 (+3376083us) ts_tablet_manager.cc:1151] Deleting tablet d76410afd7ba4c73836f4791ef739056
1024 19:48:06.523321 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.524068 (+ 747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3375945}
W20251024 19:48:06.526253 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet d76410afd7ba4c73836f4791ef739056 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d76410afd7ba4c73836f4791ef739056 already in progress: creating tablet
I20251024 19:48:06.527315 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 397) took 3264 ms. Trace:
I20251024 19:48:06.527519 1775 rpcz_store.cc:276] 1024 19:48:03.262947 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.263133 (+ 186us) service_pool.cc:225] Handling call
1024 19:48:06.527055 (+3263922us) ts_tablet_manager.cc:1151] Deleting tablet d840888183ec4ef7bb245363c5bac322
1024 19:48:06.527065 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.527243 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3263652}
W20251024 19:48:06.528163 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet d840888183ec4ef7bb245363c5bac322 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d840888183ec4ef7bb245363c5bac322 already in progress: creating tablet
I20251024 19:48:06.528883 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 396) took 3266 ms. Trace:
I20251024 19:48:06.528959 1188 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_e63d3e693e1148a7b8d2b5731b1027bc [id=5fc5a169166b436ba9343a652a650b8e] alter complete (version 1)
I20251024 19:48:06.529065 1775 rpcz_store.cc:276] 1024 19:48:03.262784 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.265632 (+ 2848us) service_pool.cc:225] Handling call
1024 19:48:06.528591 (+3262959us) ts_tablet_manager.cc:1151] Deleting tablet 9a1031d785b646b095ece9be666d2a77
1024 19:48:06.528624 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.528785 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3261254}
W20251024 19:48:06.529914 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 73278db2097d4001a538d62dacb97c9b consensus state has no leader
I20251024 19:48:06.530236 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 2960211859cd450cba83877622d88cad (DEFAULT_TABLE table=default.Table_d1c0f808bdec4b94825c72f0043421cc [id=80b38954cc9749b9aa401fbf8830d789]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.530792 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2960211859cd450cba83877622d88cad. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:06.532644 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9a1031d785b646b095ece9be666d2a77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a1031d785b646b095ece9be666d2a77 already in progress: creating tablet
I20251024 19:48:06.532985 1766 ts_tablet_manager.cc:1434] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251024 19:48:06.533303 1766 tablet_bootstrap.cc:492] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.535218 1766 tablet_bootstrap.cc:654] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.535324 1775 tablet_replica.cc:333] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:06.535833 1775 raft_consensus.cc:2243] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.536522 1775 raft_consensus.cc:2272] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.537755 1775 ts_tablet_manager.cc:1916] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.538441 1776 ts_tablet_manager.cc:1929] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:06.539841 1776 log.cc:1199] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/c04082bd1b444b468c177bef239701c6
I20251024 19:48:06.540480 1776 ts_tablet_manager.cc:1950] T c04082bd1b444b468c177bef239701c6 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:06.541903 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 458) took 1722 ms. Trace:
I20251024 19:48:06.543040 1776 rpcz_store.cc:276] 1024 19:48:04.819477 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.819517 (+ 40us) service_pool.cc:225] Handling call
1024 19:48:06.513490 (+1693973us) ts_tablet_manager.cc:1151] Deleting tablet c04082bd1b444b468c177bef239701c6
1024 19:48:06.513502 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.519207 (+ 5705us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.538302 (+ 19095us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.541862 (+ 3560us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693762}
I20251024 19:48:06.543836 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet c04082bd1b444b468c177bef239701c6 (table default.Table_24200cd6f19d460cae963afb9b7f8724 [id=a2789faf4abe4c09b0a7515d941831a6]) successfully deleted
I20251024 19:48:06.543962 1776 tablet_replica.cc:333] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:48:06.546281 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 250f106664be4b5aaa330dbb4e43cb7b (table default.Table_dfeadb429d564640af3d8d78fa50e64e [id=963d13a1679345489335e30a9aa349b1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 250f106664be4b5aaa330dbb4e43cb7b consensus state has no leader
I20251024 19:48:06.550802 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fcba3502a1824559a30dc2ffa6a9d6b0" } new_table_name: "default.Table_ace1982961c34807a364bd4f35907799" modify_external_catalogs: true
I20251024 19:48:06.544428 1776 raft_consensus.cc:2243] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.552978 1775 ts_tablet_manager.cc:1929] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.553202 1776 raft_consensus.cc:2272] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.553267 1775 log.cc:1199] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/7d1933bb728845abad5e6b09b24868f0
I20251024 19:48:06.554301 1775 ts_tablet_manager.cc:1950] T 7d1933bb728845abad5e6b09b24868f0 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:06.554715 1776 ts_tablet_manager.cc:1916] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.556154 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 395) took 3294 ms. Trace:
I20251024 19:48:06.557057 1775 rpcz_store.cc:276] 1024 19:48:03.262116 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.265872 (+ 3756us) service_pool.cc:225] Handling call
1024 19:48:06.535176 (+3269304us) ts_tablet_manager.cc:1151] Deleting tablet 7d1933bb728845abad5e6b09b24868f0
1024 19:48:06.535189 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.545828 (+ 10639us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.552705 (+ 6877us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.556091 (+ 3386us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3267513}
I20251024 19:48:06.557830 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 7d1933bb728845abad5e6b09b24868f0 (table default.Table_c564241bf74c49deaa26052b5c1d5afe [id=ec0b3dc60c1a47f0b77f5de634de0b33]) successfully deleted
I20251024 19:48:06.557828 2370 tablet.cc:1722] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251024 19:48:06.559043 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 399) took 3003 ms. Trace:
I20251024 19:48:06.559232 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a12e415590a4c71aab4ad7d30412c83" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:06.559463 1509 raft_consensus.cc:2468] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:06.560384 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c357a2a81064b3891afda6d252898b9 with delete_type TABLET_DATA_DELETED (replaced by tablet d38154d3ffd34f4b8d7305b1c23a4fae) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.560400 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet c04082bd1b444b468c177bef239701c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.560683 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fd7726c532145c6b44ea9a37544510d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.563860 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet ebcc09d8e029451d8fb6b6601ad16087 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:06.568490 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d601fa3e84af4fdc8768772e2283a86f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:06.568704 1351 tablet_service.cc:1505] Processing CreateTablet for tablet ca95e470a3db47ceb2da9246e4bc094a (DEFAULT_TABLE table=default.Table_f052d9f6feec4b35be9654644bbb59c9 [id=d3271af922bc4446a73cb679761e3bf2]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:06.569797 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d601fa3e84af4fdc8768772e2283a86f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.569875 1766 tablet_bootstrap.cc:492] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.570099 1766 ts_tablet_manager.cc:1403] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.005s
I20251024 19:48:06.571007 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 2960211859cd450cba83877622d88cad (DEFAULT_TABLE table=default.Table_d1c0f808bdec4b94825c72f0043421cc [id=80b38954cc9749b9aa401fbf8830d789]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.571161 1766 raft_consensus.cc:359] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.571322 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4b3a50db1714eb38cc0ea03f612fc95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.571420 1766 raft_consensus.cc:385] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.571522 1766 raft_consensus.cc:740] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.572297 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4c1febbedc3448728852a72f7f5d2d44" } new_table_name: "default.Table_466298cf05da40c1a6acf5e5fb65fa83" modify_external_catalogs: true
I20251024 19:48:06.574167 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8ae396f2397d425097a8f71cb1c9c910" } new_table_name: "default.Table_420ebc050cf440e9a59a18e26f7b83af" modify_external_catalogs: true
I20251024 19:48:06.574870 1308 leader_election.cc:304] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:06.573007 1766 consensus_queue.cc:260] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.575484 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6cc41f0070cf4f22b3ce19c688e77c50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:06.576081 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cc41f0070cf4f22b3ce19c688e77c50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.577178 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2960211859cd450cba83877622d88cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.578488 1766 ts_tablet_manager.cc:1434] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251024 19:48:06.578928 1766 tablet_bootstrap.cc:492] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.581058 1766 tablet_bootstrap.cc:654] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.582921 1192 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet ce0ef4443e574846a9f48fef3b893352 (table default.Table_f203b363de4c47fca0a0599140d18279 [id=ec57efb4d37e44f1b68045f58bc11c45]). Expected version 1 got 0
I20251024 19:48:06.583171 1192 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]). Expected version 1 got 0
I20251024 19:48:06.584393 1776 ts_tablet_manager.cc:1929] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:06.584653 1776 log.cc:1199] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/6ec5c7bf36874423b4790c6169da8bd0
I20251024 19:48:06.585151 1776 ts_tablet_manager.cc:1950] T 6ec5c7bf36874423b4790c6169da8bd0 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:06.586380 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 459) took 1766 ms. Trace:
I20251024 19:48:06.586576 1776 rpcz_store.cc:276] 1024 19:48:04.819880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.819916 (+ 36us) service_pool.cc:225] Handling call
1024 19:48:06.543893 (+1723977us) ts_tablet_manager.cc:1151] Deleting tablet 6ec5c7bf36874423b4790c6169da8bd0
1024 19:48:06.543903 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.566911 (+ 23008us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.584232 (+ 17321us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.586212 (+ 1980us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1723813}
I20251024 19:48:06.587255 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 6ec5c7bf36874423b4790c6169da8bd0 (table default.Table_52ecaf891a2449d78bb8a354e7fb904b [id=e0a69533d0fa41a4ad58f08e510fa39c]) successfully deleted
I20251024 19:48:06.587100 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet 73ffe342f541447d8519ee24bc98a0f3 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:06.587575 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 460) took 1650 ms. Trace:
I20251024 19:48:06.587770 1776 rpcz_store.cc:276] 1024 19:48:04.936960 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.937033 (+ 73us) service_pool.cc:225] Handling call
1024 19:48:06.587314 (+1650281us) ts_tablet_manager.cc:1151] Deleting tablet 6424b2347ef642feb38a9596f2d2be1d
1024 19:48:06.587329 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.587499 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1649949}
W20251024 19:48:06.588357 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 6424b2347ef642feb38a9596f2d2be1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6424b2347ef642feb38a9596f2d2be1d already in progress: creating tablet
I20251024 19:48:06.588632 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 461) took 1602 ms. Trace:
I20251024 19:48:06.588799 1776 rpcz_store.cc:276] 1024 19:48:04.986265 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.986334 (+ 69us) service_pool.cc:225] Handling call
1024 19:48:06.588402 (+1602068us) ts_tablet_manager.cc:1151] Deleting tablet ab488e4971874a16b2e052c84c250777
1024 19:48:06.588415 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.588546 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1601584}
I20251024 19:48:06.588876 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_099958da5fc9487db3398646189a7194" } modify_external_catalogs: true
I20251024 19:48:06.589056 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_099958da5fc9487db3398646189a7194" } modify_external_catalogs: true
W20251024 19:48:06.589448 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet ab488e4971874a16b2e052c84c250777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab488e4971874a16b2e052c84c250777 already in progress: creating tablet
I20251024 19:48:06.589599 1776 tablet_replica.cc:333] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:06.589913 1776 raft_consensus.cc:2243] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.590528 1776 raft_consensus.cc:2272] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.591558 1766 tablet_bootstrap.cc:492] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.591794 1766 ts_tablet_manager.cc:1403] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20251024 19:48:06.593084 1766 raft_consensus.cc:359] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:06.593415 1766 raft_consensus.cc:385] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.593657 1766 raft_consensus.cc:740] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.593832 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca95e470a3db47ceb2da9246e4bc094a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.594111 1766 consensus_queue.cc:260] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:06.595337 1766 ts_tablet_manager.cc:1434] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251024 19:48:06.596170 1766 tablet_bootstrap.cc:492] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.596381 2272 raft_consensus.cc:2804] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:06.596530 2272 raft_consensus.cc:493] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:06.596654 2272 raft_consensus.cc:3060] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:06.598057 1776 ts_tablet_manager.cc:1916] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.601938 1766 tablet_bootstrap.cc:654] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.602255 2371 consensus_queue.cc:1048] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251024 19:48:06.607259 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet ad055dc3387b414bb55d09b9683ba17e (table default.Table_cde3670bf0d04dc0b2049193bcee10d4 [id=c3de6937e93a48dcb88cabee59a09d32]). Expected version 1 got 0
I20251024 19:48:06.607589 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]). Expected version 1 got 0
I20251024 19:48:06.607847 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 452f77091adc4cf09de2231f2f6d845b (table default.Table_0fa2426ca885458797e98bedc44dec04 [id=9a49120c5b654676a63cd3966eaf31a9]). Expected version 3 got 0
I20251024 19:48:06.609310 2321 raft_consensus.cc:493] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:06.609524 2321 raft_consensus.cc:515] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.610363 2321 leader_election.cc:290] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:06.611387 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce0ef4443e574846a9f48fef3b893352" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:06.612186 1586 leader_election.cc:343] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.612284 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0e883c73a01f43fb99a5ad745839fae3" } modify_external_catalogs: true
I20251024 19:48:06.612509 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0e883c73a01f43fb99a5ad745839fae3" } modify_external_catalogs: true
W20251024 19:48:06.612795 2380 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 19:48:06.349781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.252.254:36505 (local address 127.0.252.193:40215)
1024 19:48:06.581693 (+231912us) negotiation.cc:107] Waiting for socket to connect
1024 19:48:06.588189 (+ 6496us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.252.254:36505: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":227190,"threads_started":1}
I20251024 19:48:06.613387 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce0ef4443e574846a9f48fef3b893352" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:06.614148 1586 leader_election.cc:343] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.614892 1586 leader_election.cc:304] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:06.615484 2321 raft_consensus.cc:2749] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:06.621884 1776 ts_tablet_manager.cc:1929] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.623188 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 2960211859cd450cba83877622d88cad (DEFAULT_TABLE table=default.Table_d1c0f808bdec4b94825c72f0043421cc [id=80b38954cc9749b9aa401fbf8830d789]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.624228 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2960211859cd450cba83877622d88cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.624233 2367 tablet.cc:1722] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251024 19:48:06.625451 1766 tablet_bootstrap.cc:492] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.625787 1766 ts_tablet_manager.cc:1403] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.002s
I20251024 19:48:06.626910 1766 raft_consensus.cc:359] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.627272 1766 raft_consensus.cc:385] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.627414 1766 raft_consensus.cc:740] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.627825 1766 consensus_queue.cc:260] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.632066 1766 ts_tablet_manager.cc:1434] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251024 19:48:06.632526 1766 tablet_bootstrap.cc:492] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.623860 1776 log.cc:1199] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/a48249f1824a40b087b80a563816d485
I20251024 19:48:06.633327 1776 ts_tablet_manager.cc:1950] T a48249f1824a40b087b80a563816d485 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:06.635617 1766 tablet_bootstrap.cc:654] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.636751 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 462) took 1573 ms. Trace:
I20251024 19:48:06.640353 2272 raft_consensus.cc:515] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.642091 1776 rpcz_store.cc:276] 1024 19:48:05.063406 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.089638 (+ 26232us) service_pool.cc:225] Handling call
1024 19:48:06.589465 (+1499827us) ts_tablet_manager.cc:1151] Deleting tablet a48249f1824a40b087b80a563816d485
1024 19:48:06.589477 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.608124 (+ 18647us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.621703 (+ 13579us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.636698 (+ 14995us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1499481}
I20251024 19:48:06.642242 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a12e415590a4c71aab4ad7d30412c83" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:06.642623 2272 leader_election.cc:290] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:06.643023 1776 tablet_replica.cc:333] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:06.643054 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet a48249f1824a40b087b80a563816d485 (table default.Table_f96cb6877d844a50aacb91514a539fee [id=d9ab6489833b4989a69c975d734bcd2f]) successfully deleted
W20251024 19:48:06.643529 1305 leader_election.cc:343] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.559335 1775 rpcz_store.cc:276] 1024 19:48:03.555185 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.567092 (+ 11907us) service_pool.cc:225] Handling call
1024 19:48:06.558734 (+2991642us) ts_tablet_manager.cc:1151] Deleting tablet 38818c7997c84cedad5f9b09eef77105
1024 19:48:06.558747 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.558945 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2983417}
I20251024 19:48:06.643399 2141 consensus_queue.cc:1048] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251024 19:48:06.644896 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a12e415590a4c71aab4ad7d30412c83" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:06.645913 1510 raft_consensus.cc:3060] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:06.648370 2272 consensus_queue.cc:1048] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251024 19:48:06.649183 1510 raft_consensus.cc:2468] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:06.649878 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ecf8ed863a5b41fb888927d73b57c8a6" } modify_external_catalogs: true
W20251024 19:48:06.649896 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 38818c7997c84cedad5f9b09eef77105 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38818c7997c84cedad5f9b09eef77105 already in progress: creating tablet
I20251024 19:48:06.650202 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ecf8ed863a5b41fb888927d73b57c8a6" } modify_external_catalogs: true
I20251024 19:48:06.650624 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 405) took 3037 ms. Trace:
I20251024 19:48:06.650902 1776 raft_consensus.cc:2243] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.651273 1308 leader_election.cc:304] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:06.652053 1776 raft_consensus.cc:2272] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.652251 1775 rpcz_store.cc:276] 1024 19:48:03.613202 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.613255 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:06.650392 (+3037137us) ts_tablet_manager.cc:1151] Deleting tablet 786fe2fd6fca407e83a76f74f96a8b16
1024 19:48:06.650407 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.650568 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3030940}
I20251024 19:48:06.653317 1776 ts_tablet_manager.cc:1916] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:06.653739 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 786fe2fd6fca407e83a76f74f96a8b16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 786fe2fd6fca407e83a76f74f96a8b16 already in progress: creating tablet
I20251024 19:48:06.654441 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 404) took 3042 ms. Trace:
I20251024 19:48:06.654716 1775 rpcz_store.cc:276] 1024 19:48:03.611545 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.611677 (+ 132us) service_pool.cc:225] Handling call
1024 19:48:06.654260 (+3042583us) ts_tablet_manager.cc:1151] Deleting tablet 701c93a600964c20a275f5942a2ff863
1024 19:48:06.654273 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.654396 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3034845}
I20251024 19:48:06.660214 1775 tablet_replica.cc:333] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:06.660597 1775 raft_consensus.cc:2243] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.660892 1775 raft_consensus.cc:2272] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.661749 1775 ts_tablet_manager.cc:1916] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.665218 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 86ff78a220714c4f9ac79f82b44597f6 (DEFAULT_TABLE table=default.Table_3bc26ab1c8fd48c682794066b1d61d4e [id=9381a685e2e54e20ba6a7074643de773]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.665845 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 86ff78a220714c4f9ac79f82b44597f6 (DEFAULT_TABLE table=default.Table_3bc26ab1c8fd48c682794066b1d61d4e [id=9381a685e2e54e20ba6a7074643de773]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.666221 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ff78a220714c4f9ac79f82b44597f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.666373 1766 tablet_bootstrap.cc:492] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.666576 1766 ts_tablet_manager.cc:1403] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
W20251024 19:48:06.667749 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 27f1a5eccdf6430c9e220988043596b0 consensus state has no leader
I20251024 19:48:06.667671 1766 raft_consensus.cc:359] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.667886 1766 raft_consensus.cc:385] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.667831 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 86ff78a220714c4f9ac79f82b44597f6 (DEFAULT_TABLE table=default.Table_3bc26ab1c8fd48c682794066b1d61d4e [id=9381a685e2e54e20ba6a7074643de773]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.667977 1766 raft_consensus.cc:740] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.668118 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6a5150d7236d4d558c9f9fb4e70a78dd" } modify_external_catalogs: true
I20251024 19:48:06.668219 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ff78a220714c4f9ac79f82b44597f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.668241 1766 consensus_queue.cc:260] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.668318 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6a5150d7236d4d558c9f9fb4e70a78dd" } modify_external_catalogs: true
I20251024 19:48:06.668853 1766 ts_tablet_manager.cc:1434] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:06.669133 1766 tablet_bootstrap.cc:492] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.669129 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 4cac515c0a00403fb680454c1721be0a (DEFAULT_TABLE table=default.Table_14c1f0af1332491a8ad3d0a6252f840d [id=5ceec89ebd754a46a3b16385129d9958]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.669615 2272 raft_consensus.cc:2804] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:06.670001 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cac515c0a00403fb680454c1721be0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.671046 1766 tablet_bootstrap.cc:654] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.671358 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ff78a220714c4f9ac79f82b44597f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.673417 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 4cac515c0a00403fb680454c1721be0a (DEFAULT_TABLE table=default.Table_14c1f0af1332491a8ad3d0a6252f840d [id=5ceec89ebd754a46a3b16385129d9958]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.673975 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cac515c0a00403fb680454c1721be0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.684670 1776 ts_tablet_manager.cc:1929] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.691537 1776 log.cc:1199] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/b5473699c1b647f8b86ee43fdb56bdfd
I20251024 19:48:06.692499 1776 ts_tablet_manager.cc:1950] T b5473699c1b647f8b86ee43fdb56bdfd P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:06.694136 1775 ts_tablet_manager.cc:1929] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:06.697270 2272 raft_consensus.cc:697] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:06.697194 1766 tablet_bootstrap.cc:492] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.697729 1766 ts_tablet_manager.cc:1403] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.003s
I20251024 19:48:06.698952 1766 raft_consensus.cc:359] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.699290 1766 raft_consensus.cc:385] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.699430 1766 raft_consensus.cc:740] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.700002 1766 consensus_queue.cc:260] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.701056 1766 ts_tablet_manager.cc:1434] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251024 19:48:06.701802 1775 log.cc:1199] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/346383e85cc64b3a8da70a71868ceccc
I20251024 19:48:06.701758 2272 consensus_queue.cc:237] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.697371 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 463) took 1593 ms. Trace:
I20251024 19:48:06.702756 1775 ts_tablet_manager.cc:1950] T 346383e85cc64b3a8da70a71868ceccc P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:06.702906 1776 rpcz_store.cc:276] 1024 19:48:05.104024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.104095 (+ 71us) service_pool.cc:225] Handling call
1024 19:48:06.642929 (+1538834us) ts_tablet_manager.cc:1151] Deleting tablet b5473699c1b647f8b86ee43fdb56bdfd
1024 19:48:06.642944 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.680046 (+ 37102us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.684534 (+ 4488us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.697297 (+ 12763us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1538455}
I20251024 19:48:06.703557 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet b5473699c1b647f8b86ee43fdb56bdfd (table default.Table_c700d3a5bb0c4963ace8624ae2ef882d [id=30691828be584171b7ee2472c9a25b4b]) successfully deleted
I20251024 19:48:06.703665 1776 tablet_replica.cc:333] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:06.703984 1776 raft_consensus.cc:2243] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.704279 1776 raft_consensus.cc:2272] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.704388 1767 tablet_bootstrap.cc:492] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:06.704648 1767 ts_tablet_manager.cc:1403] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.216s user 0.034s sys 0.084s
I20251024 19:48:06.705483 1767 raft_consensus.cc:359] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.705698 2321 raft_consensus.cc:493] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:06.705782 1767 raft_consensus.cc:385] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.705891 2321 raft_consensus.cc:515] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.705300 1776 ts_tablet_manager.cc:1916] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.706578 2321 leader_election.cc:290] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:06.705881 1767 raft_consensus.cc:740] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.706916 1767 consensus_queue.cc:260] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.707713 1767 ts_tablet_manager.cc:1434] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:06.708105 1767 tablet_bootstrap.cc:492] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:06.708256 1774 ts_tablet_manager.cc:1929] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.709987 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fd7726c532145c6b44ea9a37544510d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:06.710260 1767 tablet_bootstrap.cc:654] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:06.710649 1586 leader_election.cc:343] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.711479 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fd7726c532145c6b44ea9a37544510d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:06.712215 1586 leader_election.cc:343] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.712357 1586 leader_election.cc:304] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:06.712649 2321 raft_consensus.cc:2749] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:06.716326 1766 tablet_bootstrap.cc:492] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.719921 1766 tablet_bootstrap.cc:654] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:06.722491 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 196848us
I20251024 19:48:06.722714 1774 log.cc:1199] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/7d1933bb728845abad5e6b09b24868f0
I20251024 19:48:06.724711 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 4cac515c0a00403fb680454c1721be0a (DEFAULT_TABLE table=default.Table_14c1f0af1332491a8ad3d0a6252f840d [id=5ceec89ebd754a46a3b16385129d9958]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.725134 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cac515c0a00403fb680454c1721be0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.725139 1774 ts_tablet_manager.cc:1950] T 7d1933bb728845abad5e6b09b24868f0 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:06.726569 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_aa4ebf0155944ee2bfba2c23eacb5655" } new_table_name: "default.Table_60d5ed8e9b3846a8bc33ffae6069dda2" modify_external_catalogs: true
I20251024 19:48:06.728792 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 408) took 3017 ms. Trace:
I20251024 19:48:06.728984 1775 rpcz_store.cc:276] 1024 19:48:03.711060 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.711750 (+ 690us) service_pool.cc:225] Handling call
1024 19:48:06.660127 (+2948377us) ts_tablet_manager.cc:1151] Deleting tablet 346383e85cc64b3a8da70a71868ceccc
1024 19:48:06.660139 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.687279 (+ 27140us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.693993 (+ 6714us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.712967 (+ 18974us) spinlock_profiling.cc:243] Waited 8.8 ms on lock 0x61800000d518. stack: 00007f26c12049f6 00007f26c12047b0 00007f26c01e03ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f26c11ed2fd 00007f26c11ee36e 00007f26d4e20c9d 00007f26d4d9f203 00007f26d0fd46f3 00007f26d10072d4 00007f26d0fe7d70 00007f26d0fe7afd 0000000000a8867c 00007f26c124ad7f
1024 19:48:06.728678 (+ 15711us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20248832,"tablet-delete.queue_time_us":2947885}
I20251024 19:48:06.729349 1767 tablet_bootstrap.cc:492] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:06.729504 1767 ts_tablet_manager.cc:1403] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.005s
I20251024 19:48:06.729696 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 346383e85cc64b3a8da70a71868ceccc (table default.Table_acb41c25b40742ef92f4bb9d45ada0a3 [id=44c9c172ae1242d3a7ac272857209ff8]) successfully deleted
I20251024 19:48:06.730341 1767 raft_consensus.cc:359] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.730535 1767 raft_consensus.cc:385] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.730633 1767 raft_consensus.cc:740] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.730908 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet aebf76d4a66947ed9756643740418798 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.730875 1767 consensus_queue.cc:260] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.731155 1775 tablet_replica.cc:333] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:06.731577 1767 ts_tablet_manager.cc:1434] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:06.731546 1775 raft_consensus.cc:2243] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.731963 1767 tablet_bootstrap.cc:492] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:06.735589 1767 tablet_bootstrap.cc:654] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.735584 1477 tablet_service.cc:1505] Processing CreateTablet for tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 (DEFAULT_TABLE table=default.Table_928046849b864c76b36b8a2526f9bc5e [id=b54040c991354937a9ba3afdd9a75470]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.736032 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.736625 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 5368789df6024d7daa9ccc2257137d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.738096 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 98dd8bacd8374f469799872859db4206 (DEFAULT_TABLE table=default.Table_9db467b0e3b745678279c3f288a7c988 [id=43ad5dd683d6422286b6a5c2a96e1630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.738498 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98dd8bacd8374f469799872859db4206. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.741643 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 98dd8bacd8374f469799872859db4206 (DEFAULT_TABLE table=default.Table_9db467b0e3b745678279c3f288a7c988 [id=43ad5dd683d6422286b6a5c2a96e1630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.742040 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98dd8bacd8374f469799872859db4206. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.743093 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 7ecf3276437a43cf84a47a0e66553e0a (DEFAULT_TABLE table=default.Table_9aa983432fce4338b09f4182277c9825 [id=20259d1e36e04ef8b8ea6e8e64cceaa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.744028 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ecf3276437a43cf84a47a0e66553e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.744786 1624 tablet_service.cc:1505] Processing CreateTablet for tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 (DEFAULT_TABLE table=default.Table_928046849b864c76b36b8a2526f9bc5e [id=b54040c991354937a9ba3afdd9a75470]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.745086 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 7ecf3276437a43cf84a47a0e66553e0a (DEFAULT_TABLE table=default.Table_9aa983432fce4338b09f4182277c9825 [id=20259d1e36e04ef8b8ea6e8e64cceaa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.745213 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.745441 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ecf3276437a43cf84a47a0e66553e0a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:06.749524 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3da81c18d9264f38966944ad13f256d8 consensus state has no leader
I20251024 19:48:06.752452 1188 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet b20a6845ae1645c0b51f096d471d4ccc (table default.Table_50e12cf5f39d4fa2983835ef0dbbc3d6 [id=201e5232763a4940a1288a56a695f71c]). Expected version 1 got 0
I20251024 19:48:06.752668 1188 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet ad055dc3387b414bb55d09b9683ba17e (table default.Table_cde3670bf0d04dc0b2049193bcee10d4 [id=c3de6937e93a48dcb88cabee59a09d32]). Expected version 1 got 0
I20251024 19:48:06.753798 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 27f1a5eccdf6430c9e220988043596b0 (DEFAULT_TABLE table=default.Table_ace1982961c34807a364bd4f35907799 [id=82b0aacd36a94b15b6ceb396bd9eba1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.754230 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27f1a5eccdf6430c9e220988043596b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.755870 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 362) took 3509 ms. Trace:
I20251024 19:48:06.757409 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 27f1a5eccdf6430c9e220988043596b0 (DEFAULT_TABLE table=default.Table_ace1982961c34807a364bd4f35907799 [id=82b0aacd36a94b15b6ceb396bd9eba1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.757827 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27f1a5eccdf6430c9e220988043596b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.759203 1767 tablet_bootstrap.cc:492] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:06.759359 1767 ts_tablet_manager.cc:1403] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20251024 19:48:06.759721 1774 rpcz_store.cc:276] 1024 19:48:03.216992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.217046 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:06.292898 (+3075852us) ts_tablet_manager.cc:1151] Deleting tablet 7d1933bb728845abad5e6b09b24868f0
1024 19:48:06.292909 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.556962 (+264053us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.708131 (+151169us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.725934 (+ 17803us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":135402,"tablet-delete.queue_time_us":3075665}
I20251024 19:48:06.760121 1767 raft_consensus.cc:359] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.760308 1767 raft_consensus.cc:385] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.760303 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 7d1933bb728845abad5e6b09b24868f0 (table default.Table_c564241bf74c49deaa26052b5c1d5afe [id=ec0b3dc60c1a47f0b77f5de634de0b33]) successfully deleted
I20251024 19:48:06.760371 1767 raft_consensus.cc:740] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.760640 1767 consensus_queue.cc:260] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.761147 1767 ts_tablet_manager.cc:1434] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:06.761191 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 925c7cd6dab4465aa86eb3ba20e9618e (DEFAULT_TABLE table=default.Table_747b358db68e43159e1616089b57313a [id=f806e678fdab42489e4c0f88425b566c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.761514 1767 tablet_bootstrap.cc:492] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:06.761669 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 925c7cd6dab4465aa86eb3ba20e9618e (DEFAULT_TABLE table=default.Table_747b358db68e43159e1616089b57313a [id=f806e678fdab42489e4c0f88425b566c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.762020 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925c7cd6dab4465aa86eb3ba20e9618e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.763958 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 87575045817948d9b4980944b2e62614 (DEFAULT_TABLE table=default.Table_94d237360e434747ba2dcdf09d4367cc [id=f716a503131141bcbd738a745218f7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.764155 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925c7cd6dab4465aa86eb3ba20e9618e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.757527 1346 tablet_service.cc:1505] Processing CreateTablet for tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 (DEFAULT_TABLE table=default.Table_928046849b864c76b36b8a2526f9bc5e [id=b54040c991354937a9ba3afdd9a75470]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.764825 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87575045817948d9b4980944b2e62614. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.764462 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 7ecf3276437a43cf84a47a0e66553e0a (DEFAULT_TABLE table=default.Table_9aa983432fce4338b09f4182277c9825 [id=20259d1e36e04ef8b8ea6e8e64cceaa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.767578 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 4b39e014607b4437b255c26c92c71861 (DEFAULT_TABLE table=default.Table_74e4f24f12db403686548924c83c5191 [id=508d6cdc5d4a494bad31eaf36383e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.767853 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 87575045817948d9b4980944b2e62614 (DEFAULT_TABLE table=default.Table_94d237360e434747ba2dcdf09d4367cc [id=f716a503131141bcbd738a745218f7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.768146 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 363) took 3546 ms. Trace:
I20251024 19:48:06.769651 1767 tablet_bootstrap.cc:654] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.769708 1774 rpcz_store.cc:276] 1024 19:48:03.217985 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.218048 (+ 63us) service_pool.cc:225] Handling call
1024 19:48:06.760128 (+3542080us) ts_tablet_manager.cc:1151] Deleting tablet d840888183ec4ef7bb245363c5bac322
1024 19:48:06.760141 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.764410 (+ 4269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3541838}
W20251024 19:48:06.771126 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet d840888183ec4ef7bb245363c5bac322 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d840888183ec4ef7bb245363c5bac322 already in progress: creating tablet
I20251024 19:48:06.771373 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 925c7cd6dab4465aa86eb3ba20e9618e (DEFAULT_TABLE table=default.Table_747b358db68e43159e1616089b57313a [id=f806e678fdab42489e4c0f88425b566c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.771509 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 87575045817948d9b4980944b2e62614 (DEFAULT_TABLE table=default.Table_94d237360e434747ba2dcdf09d4367cc [id=f716a503131141bcbd738a745218f7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.771768 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925c7cd6dab4465aa86eb3ba20e9618e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.772022 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87575045817948d9b4980944b2e62614. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.772184 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 4b39e014607b4437b255c26c92c71861 (DEFAULT_TABLE table=default.Table_74e4f24f12db403686548924c83c5191 [id=508d6cdc5d4a494bad31eaf36383e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.772435 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 4b39e014607b4437b255c26c92c71861 (DEFAULT_TABLE table=default.Table_74e4f24f12db403686548924c83c5191 [id=508d6cdc5d4a494bad31eaf36383e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.772549 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b39e014607b4437b255c26c92c71861. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.772830 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b39e014607b4437b255c26c92c71861. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.776064 2020 raft_consensus.cc:493] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:06.776218 2020 raft_consensus.cc:515] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.777022 2020 leader_election.cc:290] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:06.778182 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3754540ae21e4c65b5af3d0aa3f9189f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:06.778359 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 7eb62791100f477784ab315923b6ceb6 (DEFAULT_TABLE table=default.Table_c8e3fdadfb964ad6b614c60e5d2695c5 [id=07e4ba26c7e445358bc87238aa9dcf92]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.778779 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb62791100f477784ab315923b6ceb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.779058 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 7eb62791100f477784ab315923b6ceb6 (DEFAULT_TABLE table=default.Table_c8e3fdadfb964ad6b614c60e5d2695c5 [id=07e4ba26c7e445358bc87238aa9dcf92]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:06.781651 1444 leader_election.cc:343] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.781847 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb62791100f477784ab315923b6ceb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.787583 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 6bebcdb80476423b8e6c750f749a1771 (DEFAULT_TABLE table=default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.787973 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 6bebcdb80476423b8e6c750f749a1771 (DEFAULT_TABLE table=default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.788153 1339 tablet_service.cc:1505] Processing CreateTablet for tablet bc61fded84b24e00b299e5d342076708 (DEFAULT_TABLE table=default.Table_adcb658ebf79468fb55f7635cf442d29 [id=eca5f2d498dc459ebe88f73ee0fbb1b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.788341 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bebcdb80476423b8e6c750f749a1771. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.789924 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 27f1a5eccdf6430c9e220988043596b0 (DEFAULT_TABLE table=default.Table_ace1982961c34807a364bd4f35907799 [id=82b0aacd36a94b15b6ceb396bd9eba1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.790225 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27f1a5eccdf6430c9e220988043596b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.791870 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a86db858d09749a9a4706c761c20ee9a" } modify_external_catalogs: true
I20251024 19:48:06.791992 1192 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_1865562320ef412799f93a15c66ade0c [id=aadca5a1b6d4494aa37f8a8cc76ad12a] alter complete (version 6)
I20251024 19:48:06.792107 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a86db858d09749a9a4706c761c20ee9a" } modify_external_catalogs: true
I20251024 19:48:06.793407 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0947a6b976074a85abef51497e84b52e" } new_table_name: "default.Table_7013504a3c6a4813991ac0e9022f53e1" modify_external_catalogs: true
W20251024 19:48:06.791875 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41176885447e42409ee4363bb4d903f2 consensus state has no leader
I20251024 19:48:06.794854 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_915650e3f7984a0d81ae2f98d2b1d0a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:06.795329 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_915650e3f7984a0d81ae2f98d2b1d0a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.795655 1618 tablet_service.cc:1505] Processing CreateTablet for tablet bc61fded84b24e00b299e5d342076708 (DEFAULT_TABLE table=default.Table_adcb658ebf79468fb55f7635cf442d29 [id=eca5f2d498dc459ebe88f73ee0fbb1b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.796030 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc61fded84b24e00b299e5d342076708. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.797928 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet d76410afd7ba4c73836f4791ef739056 with delete_type TABLET_DATA_DELETED (replaced by tablet b153f75caebb4d869e77529c9c9bb50d) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.798415 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec5c7bf36874423b4790c6169da8bd0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7712392f200842c4b728e019d6a3cfb0) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.800679 2321 consensus_queue.cc:1048] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:06.800923 1768 tablet_bootstrap.cc:492] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:06.803726 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a12e415590a4c71aab4ad7d30412c83 with delete_type TABLET_DATA_DELETED (replaced by tablet d6de2c21796f4bc4904f88b654c3baec) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.804850 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f88390f2534e4799a6eaa56aa4245e97
I20251024 19:48:06.805670 1775 raft_consensus.cc:2272] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.805971 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet f88390f2534e4799a6eaa56aa4245e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.806368 1775 ts_tablet_manager.cc:1916] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.806423 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fffc42ddda14bd28800917ff2f0ced5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.806684 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet ca0527af718f494a8bbcc66e7b325bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.806715 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b9fe85eed58142c1a663db71a1f34756"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:06.806914 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet f88390f2534e4799a6eaa56aa4245e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:06.807396 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9fe85eed58142c1a663db71a1f34756 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.808389 1185 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 926411b9802b43db8c57bd62e2d91924
I20251024 19:48:06.808710 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_27407bc1a5e3473abd1a59db49046843"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:06.809206 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27407bc1a5e3473abd1a59db49046843 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.809206 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fffc42ddda14bd28800917ff2f0ced5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.809973 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet b6661b9e9a174849b6093d09cd8c1251 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:06.810395 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet f88390f2534e4799a6eaa56aa4245e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:06.811036 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4798dd4a61224784b9963dd87de7a19a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:06.811663 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4798dd4a61224784b9963dd87de7a19a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.811939 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.812937 1474 tablet_service.cc:1505] Processing CreateTablet for tablet bc61fded84b24e00b299e5d342076708 (DEFAULT_TABLE table=default.Table_adcb658ebf79468fb55f7635cf442d29 [id=eca5f2d498dc459ebe88f73ee0fbb1b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.734452 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 98dd8bacd8374f469799872859db4206 (DEFAULT_TABLE table=default.Table_9db467b0e3b745678279c3f288a7c988 [id=43ad5dd683d6422286b6a5c2a96e1630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.813648 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98dd8bacd8374f469799872859db4206. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.814157 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc61fded84b24e00b299e5d342076708. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.814445 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 7eb62791100f477784ab315923b6ceb6 (DEFAULT_TABLE table=default.Table_c8e3fdadfb964ad6b614c60e5d2695c5 [id=07e4ba26c7e445358bc87238aa9dcf92]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.815546 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 7712392f200842c4b728e019d6a3cfb0 (DEFAULT_TABLE table=default.Table_52ecaf891a2449d78bb8a354e7fb904b [id=e0a69533d0fa41a4ad58f08e510fa39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.816002 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7712392f200842c4b728e019d6a3cfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.815934 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 7712392f200842c4b728e019d6a3cfb0 (DEFAULT_TABLE table=default.Table_52ecaf891a2449d78bb8a354e7fb904b [id=e0a69533d0fa41a4ad58f08e510fa39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.818029 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bebcdb80476423b8e6c750f749a1771. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.822525 2321 raft_consensus.cc:493] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:06.822683 2321 raft_consensus.cc:515] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.823362 2321 leader_election.cc:290] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:06.824229 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad055dc3387b414bb55d09b9683ba17e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:06.824537 1510 raft_consensus.cc:2468] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:06.824947 1586 leader_election.cc:304] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:06.825237 2321 raft_consensus.cc:2804] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:06.825356 2321 raft_consensus.cc:493] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:06.825477 2321 raft_consensus.cc:3060] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:06.825592 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7712392f200842c4b728e019d6a3cfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.826692 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad055dc3387b414bb55d09b9683ba17e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:06.827198 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb62791100f477784ab315923b6ceb6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:06.828055 1586 leader_election.cc:343] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.823521 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 364) took 3566 ms. Trace:
I20251024 19:48:06.828964 2321 raft_consensus.cc:515] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.829352 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc61fded84b24e00b299e5d342076708. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.829849 2321 leader_election.cc:290] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:06.830787 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 392a410845a04a439fd3e2950a2c1a7b (DEFAULT_TABLE table=default.Table_564c9b80558d45a084b6a36dc542f1d9 [id=815e37f87e3145888ac49e8b7f0f4f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.831182 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 392a410845a04a439fd3e2950a2c1a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.831334 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad055dc3387b414bb55d09b9683ba17e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:06.831537 1510 raft_consensus.cc:3060] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:06.834036 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad055dc3387b414bb55d09b9683ba17e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:06.834684 1586 leader_election.cc:343] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.836908 1510 raft_consensus.cc:2468] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:06.837553 1586 leader_election.cc:304] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:06.838349 2321 raft_consensus.cc:2804] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:06.840593 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 2c96c11697514af6bf4e80a67aadf748 (DEFAULT_TABLE table=default.Table_16e10ff29f654c72802003f0c316b89f [id=26aff5b1fa544fed9997f36a29ef392d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.840749 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 6c887b0df24c45d894ca10d2eda71444 (DEFAULT_TABLE table=default.Table_24a6bf47782a48f98a76f6bcda40352e [id=44f04e7fd5024b44ba419c472571a263]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.841095 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c96c11697514af6bf4e80a67aadf748. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.841333 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 42bade6316ab4ce88c128d365ad9fa71 (DEFAULT_TABLE table=default.Table_890d1e70131f42a1a8b2c4870dd03e7b [id=616d84c684a946e5ad966e5f0a90ec42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.841411 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 42bade6316ab4ce88c128d365ad9fa71 (DEFAULT_TABLE table=default.Table_890d1e70131f42a1a8b2c4870dd03e7b [id=616d84c684a946e5ad966e5f0a90ec42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.841706 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bade6316ab4ce88c128d365ad9fa71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.841784 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bade6316ab4ce88c128d365ad9fa71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.845893 1485 tablet_service.cc:1505] Processing CreateTablet for tablet c4d261f5f55949159aa1b72797e20fd5 (DEFAULT_TABLE table=default.Table_bc114bace73742c19c6d263e61d2bcdc [id=128d01eb3a9c4e8f9ab92b7a0db29be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.846344 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d261f5f55949159aa1b72797e20fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.848475 2020 raft_consensus.cc:493] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:06.848721 2020 raft_consensus.cc:515] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.849423 2020 leader_election.cc:290] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:06.849720 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c887b0df24c45d894ca10d2eda71444. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:06.851301 1444 leader_election.cc:343] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 6ec5c7bf36874423b4790c6169da8bd0
I20251024 19:48:06.854815 1622 tablet_service.cc:1505] Processing CreateTablet for tablet c4d261f5f55949159aa1b72797e20fd5 (DEFAULT_TABLE table=default.Table_bc114bace73742c19c6d263e61d2bcdc [id=128d01eb3a9c4e8f9ab92b7a0db29be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.855398 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d261f5f55949159aa1b72797e20fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.860199 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet b6661b9e9a174849b6093d09cd8c1251 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:06.860949 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 7712392f200842c4b728e019d6a3cfb0 (DEFAULT_TABLE table=default.Table_52ecaf891a2449d78bb8a354e7fb904b [id=e0a69533d0fa41a4ad58f08e510fa39c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.861558 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7712392f200842c4b728e019d6a3cfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.868800 2020 raft_consensus.cc:493] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:06.868953 2020 raft_consensus.cc:515] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.869844 2020 leader_election.cc:290] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:06.870695 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db17d9aff6542b6b94c09f872a4cceb" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:06.871914 1444 leader_election.cc:343] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.872937 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 6c887b0df24c45d894ca10d2eda71444 (DEFAULT_TABLE table=default.Table_24a6bf47782a48f98a76f6bcda40352e [id=44f04e7fd5024b44ba419c472571a263]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.874588 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 392a410845a04a439fd3e2950a2c1a7b (DEFAULT_TABLE table=default.Table_564c9b80558d45a084b6a36dc542f1d9 [id=815e37f87e3145888ac49e8b7f0f4f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.875288 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 392a410845a04a439fd3e2950a2c1a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.881624 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c887b0df24c45d894ca10d2eda71444. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.821933 2393 tablet.cc:1722] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251024 19:48:06.884943 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 61594.6us
I20251024 19:48:06.885375 1774 rpcz_store.cc:276] 1024 19:48:03.218781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.218902 (+ 121us) service_pool.cc:225] Handling call
1024 19:48:06.770383 (+3551481us) ts_tablet_manager.cc:1151] Deleting tablet 9a1031d785b646b095ece9be666d2a77
1024 19:48:06.770424 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.784752 (+ 14328us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3551226}
I20251024 19:48:06.885629 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3754540ae21e4c65b5af3d0aa3f9189f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:06.890909 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 2c96c11697514af6bf4e80a67aadf748 (DEFAULT_TABLE table=default.Table_16e10ff29f654c72802003f0c316b89f [id=26aff5b1fa544fed9997f36a29ef392d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.892910 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c96c11697514af6bf4e80a67aadf748. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.895998 2020 consensus_queue.cc:1048] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:06.896461 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87575045817948d9b4980944b2e62614. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.897363 1776 ts_tablet_manager.cc:1929] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.897487 1776 log.cc:1199] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/7d1933bb728845abad5e6b09b24868f0
I20251024 19:48:06.898397 1776 ts_tablet_manager.cc:1950] T 7d1933bb728845abad5e6b09b24868f0 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:06.899873 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ec5c7bf36874423b4790c6169da8bd0" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:06.903770 1768 ts_tablet_manager.cc:1403] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.392s user 0.001s sys 0.013s
W20251024 19:48:06.905516 2401 log.cc:927] Time spent T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20251024 19:48:06.906586 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 392a410845a04a439fd3e2950a2c1a7b (DEFAULT_TABLE table=default.Table_564c9b80558d45a084b6a36dc542f1d9 [id=815e37f87e3145888ac49e8b7f0f4f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.907172 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 367) took 3372 ms. Trace:
I20251024 19:48:06.907357 1774 rpcz_store.cc:276] 1024 19:48:03.521615 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.521671 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:06.888564 (+3366893us) ts_tablet_manager.cc:1151] Deleting tablet 38818c7997c84cedad5f9b09eef77105
1024 19:48:06.888576 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.894107 (+ 5531us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3366545}
I20251024 19:48:06.907970 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 372) took 3316 ms. Trace:
I20251024 19:48:06.908141 1774 rpcz_store.cc:276] 1024 19:48:03.591563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.591620 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:06.907792 (+3316172us) ts_tablet_manager.cc:1151] Deleting tablet 701c93a600964c20a275f5942a2ff863
1024 19:48:06.907801 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.907912 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3315954}
I20251024 19:48:06.908747 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b39e014607b4437b255c26c92c71861. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.908850 1766 tablet_bootstrap.cc:492] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.909610 2031 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:48:06.909902 2031 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: This master's current role is: LEADER
I20251024 19:48:06.911530 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ecf3276437a43cf84a47a0e66553e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.912674 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 392a410845a04a439fd3e2950a2c1a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.913412 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_69f805dddbf3444cb3578c026af0da38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:06.916548 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69f805dddbf3444cb3578c026af0da38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:06.917695 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db17d9aff6542b6b94c09f872a4cceb" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:06.917965 1369 raft_consensus.cc:2468] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:06.919440 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 373) took 3326 ms. Trace:
I20251024 19:48:06.919605 1774 rpcz_store.cc:276] 1024 19:48:03.593268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.593320 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:06.908627 (+3315307us) ts_tablet_manager.cc:1151] Deleting tablet 786fe2fd6fca407e83a76f74f96a8b16
1024 19:48:06.908635 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.919372 (+ 10737us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3315059}
I20251024 19:48:06.920159 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 374) took 3282 ms. Trace:
I20251024 19:48:06.920301 1774 rpcz_store.cc:276] 1024 19:48:03.637550 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.637800 (+ 250us) service_pool.cc:225] Handling call
1024 19:48:06.920017 (+3282217us) ts_tablet_manager.cc:1151] Deleting tablet ee3c521876324808857f7724028c895f
1024 19:48:06.920026 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.920114 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3268226}
I20251024 19:48:06.920681 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 375) took 3224 ms. Trace:
I20251024 19:48:06.920800 1774 rpcz_store.cc:276] 1024 19:48:03.695808 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.697318 (+ 1510us) service_pool.cc:225] Handling call
1024 19:48:06.920558 (+3223240us) ts_tablet_manager.cc:1151] Deleting tablet ee68243b4a664575b679950603bb250e
1024 19:48:06.920566 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.920636 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3223035}
I20251024 19:48:06.921115 1774 tablet_replica.cc:333] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:06.921363 1774 raft_consensus.cc:2243] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.921638 1774 raft_consensus.cc:2272] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.922649 1774 ts_tablet_manager.cc:1916] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.922730 1766 ts_tablet_manager.cc:1403] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.207s user 0.009s sys 0.002s
I20251024 19:48:06.924665 1766 raft_consensus.cc:359] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.927165 1766 raft_consensus.cc:385] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.927371 1766 raft_consensus.cc:740] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.914083 1775 ts_tablet_manager.cc:1929] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.927855 1775 log.cc:1199] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/897db285f5c84930a5b438be1cb4eb96
W20251024 19:48:06.928777 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9a1031d785b646b095ece9be666d2a77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a1031d785b646b095ece9be666d2a77 already in progress: creating tablet
W20251024 19:48:06.929487 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 38818c7997c84cedad5f9b09eef77105 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38818c7997c84cedad5f9b09eef77105 already in progress: creating tablet
W20251024 19:48:06.928109 1447 leader_election.cc:343] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.929766 1447 leader_election.cc:304] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:06.930132 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 786fe2fd6fca407e83a76f74f96a8b16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 786fe2fd6fca407e83a76f74f96a8b16 already in progress: creating tablet
W20251024 19:48:06.930436 1447 consensus_peers.cc:597] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3b4e85f741849b2a6a3e8574396ff88. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:06.930665 1447 leader_election.cc:343] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.930815 1447 leader_election.cc:304] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:06.931466 2020 raft_consensus.cc:2749] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:06.931654 2020 raft_consensus.cc:2749] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:06.932440 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet ee3c521876324808857f7724028c895f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee3c521876324808857f7724028c895f already in progress: creating tablet
I20251024 19:48:06.932833 1447 leader_election.cc:304] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
W20251024 19:48:06.932978 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet ee68243b4a664575b679950603bb250e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee68243b4a664575b679950603bb250e already in progress: creating tablet
I20251024 19:48:06.933115 2141 raft_consensus.cc:2804] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:06.933211 2141 raft_consensus.cc:493] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:06.933295 2141 raft_consensus.cc:3060] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:06.934482 2423 raft_consensus.cc:493] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:06.934634 2423 raft_consensus.cc:515] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.935267 2423 leader_election.cc:290] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:06.936141 2141 raft_consensus.cc:515] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.938763 2141 leader_election.cc:290] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:06.939837 2141 raft_consensus.cc:493] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:06.940004 2141 raft_consensus.cc:515] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:06.941207 2141 leader_election.cc:290] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:06.929425 1766 consensus_queue.cc:260] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.942557 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca0527af718f494a8bbcc66e7b325bed" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:06.943068 1444 leader_election.cc:343] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.943739 1766 ts_tablet_manager.cc:1434] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20251024 19:48:06.947609 1775 ts_tablet_manager.cc:1950] T 897db285f5c84930a5b438be1cb4eb96 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:06.948874 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 409) took 3237 ms. Trace:
I20251024 19:48:06.948405 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db17d9aff6542b6b94c09f872a4cceb" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
W20251024 19:48:06.949739 1444 leader_election.cc:343] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.950555 1766 tablet_bootstrap.cc:492] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.952493 1775 rpcz_store.cc:276] 1024 19:48:03.711191 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.712062 (+ 871us) service_pool.cc:225] Handling call
1024 19:48:06.730055 (+3017993us) ts_tablet_manager.cc:1151] Deleting tablet 897db285f5c84930a5b438be1cb4eb96
1024 19:48:06.730066 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.860339 (+130273us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.913941 (+ 53602us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.948829 (+ 34888us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3017780}
I20251024 19:48:06.954646 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 897db285f5c84930a5b438be1cb4eb96 (table default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]) successfully deleted
I20251024 19:48:06.955207 1775 tablet_replica.cc:333] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:06.955574 1775 raft_consensus.cc:2243] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:06.957921 1775 raft_consensus.cc:2272] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.959429 1775 ts_tablet_manager.cc:1916] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.958688 1766 tablet_bootstrap.cc:654] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.967533 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 464) took 1772 ms. Trace:
I20251024 19:48:06.973183 1766 tablet_bootstrap.cc:492] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.973388 1766 ts_tablet_manager.cc:1403] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.006s
I20251024 19:48:06.974344 1766 raft_consensus.cc:359] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.974586 1766 raft_consensus.cc:385] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.974673 1766 raft_consensus.cc:740] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.974928 1766 consensus_queue.cc:260] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.975335 1775 ts_tablet_manager.cc:1929] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:06.975543 1775 log.cc:1199] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/a48249f1824a40b087b80a563816d485
I20251024 19:48:06.976296 1767 tablet_bootstrap.cc:492] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:06.976490 1767 ts_tablet_manager.cc:1403] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.215s user 0.041s sys 0.062s
I20251024 19:48:06.976267 1776 rpcz_store.cc:276] 1024 19:48:05.129479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.129552 (+ 73us) service_pool.cc:225] Handling call
1024 19:48:06.703558 (+1574006us) ts_tablet_manager.cc:1151] Deleting tablet 7d1933bb728845abad5e6b09b24868f0
1024 19:48:06.703570 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.717237 (+ 13667us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.897215 (+179978us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.901713 (+ 4498us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1573024}
I20251024 19:48:06.976845 1775 ts_tablet_manager.cc:1950] T a48249f1824a40b087b80a563816d485 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:06.976893 1776 tablet_replica.cc:333] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:06.977392 2141 consensus_queue.cc:1048] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:06.977389 1767 raft_consensus.cc:359] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.977563 1767 raft_consensus.cc:385] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:06.977727 1767 raft_consensus.cc:740] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:06.977425 1766 ts_tablet_manager.cc:1434] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251024 19:48:06.977939 1767 consensus_queue.cc:260] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:06.978506 1767 ts_tablet_manager.cc:1434] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251024 19:48:06.978523 1766 tablet_bootstrap.cc:492] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.978982 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 410) took 3267 ms. Trace:
I20251024 19:48:06.980410 1775 rpcz_store.cc:276] 1024 19:48:03.711292 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.712281 (+ 989us) service_pool.cc:225] Handling call
1024 19:48:06.955115 (+3242834us) ts_tablet_manager.cc:1151] Deleting tablet a48249f1824a40b087b80a563816d485
1024 19:48:06.955130 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.965331 (+ 10201us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.975208 (+ 9877us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:06.978916 (+ 3708us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3242611}
I20251024 19:48:06.981936 1767 tablet_bootstrap.cc:492] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:06.982738 2141 consensus_queue.cc:1048] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:06.984315 1776 raft_consensus.cc:2243] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.984540 1776 raft_consensus.cc:2272] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.984804 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet a48249f1824a40b087b80a563816d485 (table default.Table_f96cb6877d844a50aacb91514a539fee [id=d9ab6489833b4989a69c975d734bcd2f]) successfully deleted
I20251024 19:48:06.985201 1775 tablet_replica.cc:333] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:06.985435 1775 raft_consensus.cc:2243] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:06.985728 1775 raft_consensus.cc:2272] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.985776 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 6bebcdb80476423b8e6c750f749a1771 (DEFAULT_TABLE table=default.Table_a17ef21a2da140b4954cf16abc6d2e95 [id=b81a7d71911844c1be495ade2ceaf1d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:06.987139 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bebcdb80476423b8e6c750f749a1771. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.987582 1775 ts_tablet_manager.cc:1916] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:06.986742 1766 tablet_bootstrap.cc:654] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:06.999379 1766 tablet_bootstrap.cc:492] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:06.999624 1766 ts_tablet_manager.cc:1403] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.002s
I20251024 19:48:07.000384 1766 raft_consensus.cc:359] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.000665 1766 raft_consensus.cc:385] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.001102 1766 raft_consensus.cc:740] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.001565 1766 consensus_queue.cc:260] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.002156 1776 ts_tablet_manager.cc:1916] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.002876 1766 ts_tablet_manager.cc:1434] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:07.003162 1766 tablet_bootstrap.cc:492] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:06.923347 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 2c96c11697514af6bf4e80a67aadf748 (DEFAULT_TABLE table=default.Table_16e10ff29f654c72802003f0c316b89f [id=26aff5b1fa544fed9997f36a29ef392d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.004885 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c96c11697514af6bf4e80a67aadf748. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.006243 1766 tablet_bootstrap.cc:654] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.005290 1775 ts_tablet_manager.cc:1929] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.007128 1775 log.cc:1199] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/b5473699c1b647f8b86ee43fdb56bdfd
I20251024 19:48:07.007616 1775 ts_tablet_manager.cc:1950] T b5473699c1b647f8b86ee43fdb56bdfd P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:07.008347 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 411) took 3296 ms. Trace:
I20251024 19:48:07.008471 1775 rpcz_store.cc:276] 1024 19:48:03.711380 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.712544 (+ 1164us) service_pool.cc:225] Handling call
1024 19:48:06.985106 (+3272562us) ts_tablet_manager.cc:1151] Deleting tablet b5473699c1b647f8b86ee43fdb56bdfd
1024 19:48:06.985115 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:06.990992 (+ 5877us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.005146 (+ 14154us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.008308 (+ 3162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3272355}
I20251024 19:48:07.008957 1775 tablet_replica.cc:333] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:07.009171 1775 raft_consensus.cc:2243] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:07.009536 1775 raft_consensus.cc:2272] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:06.924785 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 6c887b0df24c45d894ca10d2eda71444 (DEFAULT_TABLE table=default.Table_24a6bf47782a48f98a76f6bcda40352e [id=44f04e7fd5024b44ba419c472571a263]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.009910 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c887b0df24c45d894ca10d2eda71444. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.010099 1775 ts_tablet_manager.cc:1916] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.015141 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet b5473699c1b647f8b86ee43fdb56bdfd (table default.Table_c700d3a5bb0c4963ace8624ae2ef882d [id=30691828be584171b7ee2472c9a25b4b]) successfully deleted
I20251024 19:48:06.926602 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 42bade6316ab4ce88c128d365ad9fa71 (DEFAULT_TABLE table=default.Table_890d1e70131f42a1a8b2c4870dd03e7b [id=616d84c684a946e5ad966e5f0a90ec42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.015705 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bade6316ab4ce88c128d365ad9fa71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.016402 1775 ts_tablet_manager.cc:1929] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.016551 1775 log.cc:1199] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/e20881e400cf4fac914b70afc79c084f
I20251024 19:48:07.016940 1775 ts_tablet_manager.cc:1950] T e20881e400cf4fac914b70afc79c084f P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:06.930944 1338 tablet_service.cc:1505] Processing CreateTablet for tablet c4d261f5f55949159aa1b72797e20fd5 (DEFAULT_TABLE table=default.Table_bc114bace73742c19c6d263e61d2bcdc [id=128d01eb3a9c4e8f9ab92b7a0db29be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.019434 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d261f5f55949159aa1b72797e20fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:06.949638 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db17d9aff6542b6b94c09f872a4cceb" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:07.021772 1368 raft_consensus.cc:3060] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.025501 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 412) took 3313 ms. Trace:
I20251024 19:48:07.025676 1775 rpcz_store.cc:276] 1024 19:48:03.711489 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.712993 (+ 1504us) service_pool.cc:225] Handling call
1024 19:48:07.008890 (+3295897us) ts_tablet_manager.cc:1151] Deleting tablet e20881e400cf4fac914b70afc79c084f
1024 19:48:07.008902 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.013342 (+ 4440us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.016290 (+ 2948us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.025417 (+ 9127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3295672}
I20251024 19:48:07.027483 1775 tablet_replica.cc:333] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:07.027949 1775 raft_consensus.cc:2243] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:07.028594 1775 raft_consensus.cc:2272] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.028575 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet e20881e400cf4fac914b70afc79c084f (table default.Table_69e288a9d6d24a5f841f62cd9a208f0f [id=969d428d45db42d18a5a9b82ec945bfb]) successfully deleted
I20251024 19:48:07.029731 1775 ts_tablet_manager.cc:1916] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.030081 2020 raft_consensus.cc:493] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.030692 2020 raft_consensus.cc:515] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.031432 2020 leader_election.cc:290] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.025544 1368 raft_consensus.cc:2468] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:07.042169 2020 raft_consensus.cc:493] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.042368 2020 raft_consensus.cc:515] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.043107 2020 leader_election.cc:290] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.043238 1775 ts_tablet_manager.cc:1929] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.044111 2020 raft_consensus.cc:493] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.044816 1447 leader_election.cc:304] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:07.044994 2020 raft_consensus.cc:515] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.045181 2334 raft_consensus.cc:2804] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:07.045400 2334 raft_consensus.cc:697] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:07.045399 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d76410afd7ba4c73836f4791ef739056" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:07.045648 2020 leader_election.cc:290] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.045789 2334 consensus_queue.cc:237] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.046077 1775 log.cc:1199] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/31647a48fb834c80b6920c756a94250d
I20251024 19:48:07.046599 1775 ts_tablet_manager.cc:1950] T 31647a48fb834c80b6920c756a94250d P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:07.047418 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 413) took 3335 ms. Trace:
I20251024 19:48:07.047554 1775 rpcz_store.cc:276] 1024 19:48:03.711589 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.712924 (+ 1335us) service_pool.cc:225] Handling call
1024 19:48:07.027379 (+3314455us) ts_tablet_manager.cc:1151] Deleting tablet 31647a48fb834c80b6920c756a94250d
1024 19:48:07.027391 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.036551 (+ 9160us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.043111 (+ 6560us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.047371 (+ 4260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3313989}
W20251024 19:48:07.047668 1447 leader_election.cc:343] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.047997 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 406) took 3345 ms. Trace:
I20251024 19:48:07.048065 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 31647a48fb834c80b6920c756a94250d (table default.Table_41507c038ecb4e8ba8aadd563d12b6bf [id=951d0421fe2346258be8476c7efbe6c9]) successfully deleted
I20251024 19:48:07.048118 1775 rpcz_store.cc:276] 1024 19:48:03.702373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.702752 (+ 379us) service_pool.cc:225] Handling call
1024 19:48:07.047840 (+3345088us) ts_tablet_manager.cc:1151] Deleting tablet ee3c521876324808857f7724028c895f
1024 19:48:07.047853 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.047951 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3319041}
I20251024 19:48:07.048507 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 414) took 3313 ms. Trace:
W20251024 19:48:07.048565 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet ee3c521876324808857f7724028c895f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee3c521876324808857f7724028c895f already in progress: creating tablet
I20251024 19:48:07.048632 1775 rpcz_store.cc:276] 1024 19:48:03.734780 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.735587 (+ 807us) service_pool.cc:225] Handling call
1024 19:48:07.048342 (+3312755us) ts_tablet_manager.cc:1151] Deleting tablet 45cff9bcc4924e0081518c284f074940
1024 19:48:07.048353 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.048477 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3307991}
I20251024 19:48:07.048939 1775 tablet_replica.cc:333] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:07.049176 1775 raft_consensus.cc:2243] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:07.049197 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 45cff9bcc4924e0081518c284f074940 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45cff9bcc4924e0081518c284f074940 already in progress: creating tablet
I20251024 19:48:07.049371 1775 raft_consensus.cc:2272] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.049621 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b20a6845ae1645c0b51f096d471d4ccc" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:07.050390 1447 leader_election.cc:343] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.053645 1775 ts_tablet_manager.cc:1916] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.060454 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5368789df6024d7daa9ccc2257137d58" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:07.061069 1447 leader_election.cc:343] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:06.983989 1768 raft_consensus.cc:359] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.061545 1768 raft_consensus.cc:385] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.061695 1768 raft_consensus.cc:740] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.062261 1768 consensus_queue.cc:260] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.063122 1768 ts_tablet_manager.cc:1434] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.159s user 0.000s sys 0.002s
I20251024 19:48:07.063717 1768 tablet_bootstrap.cc:492] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:07.067948 1775 ts_tablet_manager.cc:1929] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.068744 1775 log.cc:1199] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/857eed60ba2a4b309e86d88a0f093762
I20251024 19:48:07.069543 1775 ts_tablet_manager.cc:1950] T 857eed60ba2a4b309e86d88a0f093762 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:07.071285 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca0527af718f494a8bbcc66e7b325bed" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:07.072345 1447 leader_election.cc:343] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.072506 1447 leader_election.cc:304] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:07.072822 2334 raft_consensus.cc:2749] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:07.073948 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 415) took 3333 ms. Trace:
I20251024 19:48:07.074385 1775 rpcz_store.cc:276] 1024 19:48:03.739985 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.744389 (+ 4404us) service_pool.cc:225] Handling call
1024 19:48:07.048865 (+3304476us) ts_tablet_manager.cc:1151] Deleting tablet 857eed60ba2a4b309e86d88a0f093762
1024 19:48:07.048873 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.057683 (+ 8810us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.067820 (+ 10137us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.073897 (+ 6077us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3263689}
I20251024 19:48:07.075120 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 857eed60ba2a4b309e86d88a0f093762 (table default.Table_b8e34cb629054ed9be6deaf4d30992a2 [id=d225099585ad4533b088821c639b5463]) successfully deleted
I20251024 19:48:07.075326 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 416) took 3243 ms. Trace:
W20251024 19:48:07.079851 2407 log.cc:927] Time spent T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
W20251024 19:48:07.080096 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 225947us
I20251024 19:48:07.081800 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b6661b9e9a174849b6093d09cd8c1251" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:07.082052 1509 raft_consensus.cc:2468] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:07.082986 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b6661b9e9a174849b6093d09cd8c1251" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:07.084745 1586 leader_election.cc:343] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.087065 2402 tablet.cc:1722] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251024 19:48:07.087388 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d76410afd7ba4c73836f4791ef739056" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:07.089890 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5368789df6024d7daa9ccc2257137d58" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:07.090746 1444 leader_election.cc:343] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.090970 1444 leader_election.cc:304] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:07.091315 2431 raft_consensus.cc:2749] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:07.091921 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b20a6845ae1645c0b51f096d471d4ccc" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:07.093081 1444 leader_election.cc:343] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.093258 1444 leader_election.cc:304] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:07.094074 2334 raft_consensus.cc:2749] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:07.094447 2334 consensus_queue.cc:1048] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:07.105399 2334 consensus_queue.cc:1048] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:07.105525 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 7d1933bb728845abad5e6b09b24868f0 (table default.Table_c564241bf74c49deaa26052b5c1d5afe [id=ec0b3dc60c1a47f0b77f5de634de0b33]) successfully deleted
I20251024 19:48:07.109005 1586 leader_election.cc:304] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:07.109908 1648 raft_consensus.cc:2468] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:07.110458 1767 tablet_bootstrap.cc:654] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.111656 2418 raft_consensus.cc:2804] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:07.111874 2418 raft_consensus.cc:493] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:07.113004 2418 raft_consensus.cc:3060] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:07.115468 1305 consensus_peers.cc:597] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5473699c1b647f8b86ee43fdb56bdfd. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:07.116446 2418 raft_consensus.cc:515] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.117497 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b6661b9e9a174849b6093d09cd8c1251" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:07.117759 1509 raft_consensus.cc:3060] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.117980 2418 leader_election.cc:290] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.120606 1509 raft_consensus.cc:2468] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:07.121018 1586 leader_election.cc:304] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:07.121323 2418 raft_consensus.cc:2804] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:07.124114 1776 ts_tablet_manager.cc:1929] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.129287 2020 raft_consensus.cc:493] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.129467 2020 raft_consensus.cc:515] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.130374 2020 leader_election.cc:290] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.124426 2418 raft_consensus.cc:697] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:07.131727 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b6661b9e9a174849b6093d09cd8c1251" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:07.132539 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fffc42ddda14bd28800917ff2f0ced5" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:07.133082 1447 leader_election.cc:343] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:07.137605 1586 leader_election.cc:343] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.138145 2418 consensus_queue.cc:237] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.142836 1776 log.cc:1199] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/50d919a0b3d74e9a97d136a84d50e24b
I20251024 19:48:07.143573 1776 ts_tablet_manager.cc:1950] T 50d919a0b3d74e9a97d136a84d50e24b P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:07.144649 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 465) took 1953 ms. Trace:
I20251024 19:48:07.145824 1776 rpcz_store.cc:276] 1024 19:48:05.191633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.191694 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:06.976814 (+1785120us) ts_tablet_manager.cc:1151] Deleting tablet 50d919a0b3d74e9a97d136a84d50e24b
1024 19:48:06.976827 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.086426 (+109599us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.123993 (+ 37567us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.144578 (+ 20585us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1784901}
I20251024 19:48:07.146476 1776 tablet_replica.cc:333] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:07.146731 1776 raft_consensus.cc:2243] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:07.146728 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 50d919a0b3d74e9a97d136a84d50e24b (table default.Table_d0ae44b1936f452d9c75bd4b888bfb30 [id=6652d9398f8e4495bdfbca917b1d0bab]) successfully deleted
I20251024 19:48:07.146749 1444 leader_election.cc:304] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:07.147114 1776 raft_consensus.cc:2272] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.147338 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fffc42ddda14bd28800917ff2f0ced5" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:07.147537 1649 raft_consensus.cc:2468] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:07.147781 1776 ts_tablet_manager.cc:1916] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.109031 2417 consensus_queue.cc:1048] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.063s
I20251024 19:48:07.153290 2020 raft_consensus.cc:2804] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:07.153395 2020 raft_consensus.cc:493] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:07.153486 2020 raft_consensus.cc:3060] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.156308 2020 raft_consensus.cc:515] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.161567 2321 raft_consensus.cc:697] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:07.161931 2321 consensus_queue.cc:237] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.164878 2020 leader_election.cc:290] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.166273 2417 consensus_queue.cc:1048] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:07.169018 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7deb97be82964ea0b2fe9caa867a6460
I20251024 19:48:07.169792 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet fc7ccf1495924782b9b6ac1173bc73a1
I20251024 19:48:07.170006 1011 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskc9NInd/build/asan/bin/kudu
/tmp/dist-test-taskc9NInd/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.252.254:36505
--webserver_interface=127.0.252.254
--webserver_port=42051
--builtin_ntp_servers=127.0.252.212:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251024 19:48:07.171926 1776 ts_tablet_manager.cc:1929] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:07.172073 1776 log.cc:1199] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/f346a90a9efe442f82750336e3893deb
I20251024 19:48:07.172030 1189 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 1d04a9b54bd6446d8deb64292cef672d (table default.Table_147e888401aa40e5a60d97884b36cb98 [id=1acdc5d0bfb94e0faf47055b20d803b0]). Expected version 1 got 0
I20251024 19:48:07.172334 2438 tablet.cc:1722] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:07.172538 1776 ts_tablet_manager.cc:1950] T f346a90a9efe442f82750336e3893deb P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:07.173373 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 469) took 1713 ms. Trace:
W20251024 19:48:07.173911 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 373aa543e1284fbc945ca802d9eddf59 consensus state has no leader
I20251024 19:48:07.174422 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5b894129f4b642909eeb0fe89ec44dea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:07.177793 2439 tablet.cc:1722] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:07.182425 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 7deb97be82964ea0b2fe9caa867a6460 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.173502 1776 rpcz_store.cc:276] 1024 19:48:05.460192 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.466385 (+ 6193us) service_pool.cc:225] Handling call
1024 19:48:07.146377 (+1679992us) ts_tablet_manager.cc:1151] Deleting tablet f346a90a9efe442f82750336e3893deb
1024 19:48:07.146387 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.151899 (+ 5512us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.171761 (+ 19862us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.173327 (+ 1566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1679761}
I20251024 19:48:07.182704 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet e20881e400cf4fac914b70afc79c084f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.182879 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 473) took 1652 ms. Trace:
I20251024 19:48:07.183002 1776 rpcz_store.cc:276] 1024 19:48:05.530793 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.530868 (+ 75us) service_pool.cc:225] Handling call
1024 19:48:07.182690 (+1651822us) ts_tablet_manager.cc:1151] Deleting tablet 15de0d8aa9e545c09233186bbe361b97
1024 19:48:07.182703 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.182834 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651505}
I20251024 19:48:07.183681 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9987a00cca5d4a8f873241da5a36b46a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:07.184011 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet fc7ccf1495924782b9b6ac1173bc73a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.184406 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 474) took 1622 ms. Trace:
I20251024 19:48:07.184813 1776 rpcz_store.cc:276] 1024 19:48:05.562230 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.562294 (+ 64us) service_pool.cc:225] Handling call
1024 19:48:07.184238 (+1621944us) ts_tablet_manager.cc:1151] Deleting tablet 3e6f8764e9ae472b9ac35a032182aed1
1024 19:48:07.184248 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.184366 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1621562}
I20251024 19:48:07.185060 1444 leader_election.cc:304] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:07.185359 1776 tablet_replica.cc:333] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:07.185801 1776 raft_consensus.cc:2243] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:07.185899 2141 consensus_queue.cc:1048] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:07.186254 2440 raft_consensus.cc:2804] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:07.186373 2440 raft_consensus.cc:493] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:07.186524 2440 raft_consensus.cc:3060] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.190251 1776 raft_consensus.cc:2272] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.190248 1775 rpcz_store.cc:276] 1024 19:48:03.831619 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.831919 (+ 300us) service_pool.cc:225] Handling call
1024 19:48:07.075162 (+3243243us) ts_tablet_manager.cc:1151] Deleting tablet 2602335327fc489fa154056c5ef5138f
1024 19:48:07.075173 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.075276 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3243048}
W20251024 19:48:07.190871 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2602335327fc489fa154056c5ef5138f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2602335327fc489fa154056c5ef5138f already in progress: creating tablet
I20251024 19:48:07.191426 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 407) took 3480 ms. Trace:
I20251024 19:48:07.191597 1776 ts_tablet_manager.cc:1916] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.191628 1775 rpcz_store.cc:276] 1024 19:48:03.710490 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.733230 (+ 22740us) service_pool.cc:225] Handling call
1024 19:48:07.191243 (+3458013us) ts_tablet_manager.cc:1151] Deleting tablet ee68243b4a664575b679950603bb250e
1024 19:48:07.191255 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.191364 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3356238}
W20251024 19:48:07.192170 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet ee68243b4a664575b679950603bb250e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee68243b4a664575b679950603bb250e already in progress: creating tablet
I20251024 19:48:07.192679 1775 tablet_replica.cc:333] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:07.192992 1775 raft_consensus.cc:2243] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:07.193810 1186 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 4b97907234e543c2806f40619046f6ae
I20251024 19:48:07.194039 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet 0266cef4e2034303978d7456c47ea98c from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:07.194646 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8916f1e163e3485db01f9d64a7db190c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:07.195436 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d610b007a0054067ab8a1278580a98d6 has no consensus state
I20251024 19:48:07.195773 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9764ce0360fd4fcfb03ab7df33ed112a" } new_table_name: "default.Table_fdf3a5f81257407d8b5d28c7f7cab1b7" modify_external_catalogs: true
I20251024 19:48:07.197750 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_11dceac7c7394c128fa1ede51ee9fdab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:07.198778 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5c5192ae905d4d4d809ebaeff38c6ca8" } new_table_name: "default.Table_9c03ce6b892d44fcbff5c92f613bc37f" modify_external_catalogs: true
I20251024 19:48:07.189530 2440 raft_consensus.cc:515] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.204893 2440 leader_election.cc:290] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.211447 1775 pending_rounds.cc:70] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e: Trying to abort 1 pending ops.
I20251024 19:48:07.211568 1775 pending_rounds.cc:77] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429325533777920 op_type: NO_OP noop_request { }
I20251024 19:48:07.211704 1775 raft_consensus.cc:2889] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:07.211879 1775 raft_consensus.cc:2272] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.212576 1775 ts_tablet_manager.cc:1916] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.212667 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet f346a90a9efe442f82750336e3893deb (table default.Table_673f2f271b3e4f9d9c5f6802a3646f8f [id=b79808d531b740d7abc0e6fc2d9ba219]) successfully deleted
W20251024 19:48:07.212935 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 15de0d8aa9e545c09233186bbe361b97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15de0d8aa9e545c09233186bbe361b97 already in progress: creating tablet
W20251024 19:48:07.213157 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3e6f8764e9ae472b9ac35a032182aed1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e6f8764e9ae472b9ac35a032182aed1 already in progress: creating tablet
I20251024 19:48:07.214015 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet fc7ccf1495924782b9b6ac1173bc73a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:07.216529 1776 ts_tablet_manager.cc:1929] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.216660 1766 tablet_bootstrap.cc:492] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:07.216688 1776 log.cc:1199] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/8a1f1892bd3b4776adc7c2519fcdf411
I20251024 19:48:07.216817 1766 ts_tablet_manager.cc:1403] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.214s user 0.015s sys 0.021s
I20251024 19:48:07.217175 1776 ts_tablet_manager.cc:1950] T 8a1f1892bd3b4776adc7c2519fcdf411 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:07.217521 1766 raft_consensus.cc:359] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.217721 1766 raft_consensus.cc:385] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.217787 1766 raft_consensus.cc:740] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.217989 1766 consensus_queue.cc:260] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.221520 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fffc42ddda14bd28800917ff2f0ced5" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:07.221748 1650 raft_consensus.cc:3060] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.222786 1766 ts_tablet_manager.cc:1434] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251024 19:48:07.223801 1766 tablet_bootstrap.cc:492] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:07.224677 1650 raft_consensus.cc:2468] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:07.225641 1766 tablet_bootstrap.cc:654] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.226332 1444 leader_election.cc:304] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:07.226696 2334 raft_consensus.cc:2804] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:07.226871 2334 raft_consensus.cc:697] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:07.227159 2334 consensus_queue.cc:237] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:07.230317 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8916f1e163e3485db01f9d64a7db190c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:07.232561 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11dceac7c7394c128fa1ede51ee9fdab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:07.233829 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b894129f4b642909eeb0fe89ec44dea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.235266 1766 tablet_bootstrap.cc:492] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:07.235445 1766 ts_tablet_manager.cc:1403] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20251024 19:48:07.236223 1766 raft_consensus.cc:359] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.236394 1766 raft_consensus.cc:385] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.236352 2440 raft_consensus.cc:493] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.236593 2440 raft_consensus.cc:515] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:07.237318 1189 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]). Expected version 1 got 0
I20251024 19:48:07.237516 2440 leader_election.cc:290] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:07.237741 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 475) took 1665 ms. Trace:
W20251024 19:48:07.237478 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9987a00cca5d4a8f873241da5a36b46a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.238498 2417 consensus_queue.cc:1048] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:07.238600 1776 rpcz_store.cc:276] 1024 19:48:05.572505 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.572564 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:07.185295 (+1612731us) ts_tablet_manager.cc:1151] Deleting tablet 8a1f1892bd3b4776adc7c2519fcdf411
1024 19:48:07.185303 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.206507 (+ 21204us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.216402 (+ 9895us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.237675 (+ 21273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1612424}
I20251024 19:48:07.239691 1776 tablet_replica.cc:333] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:07.240202 1776 raft_consensus.cc:2243] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:07.240589 1775 ts_tablet_manager.cc:1929] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.241185 1776 raft_consensus.cc:2272] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.242893 1776 ts_tablet_manager.cc:1916] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.243446 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 8a1f1892bd3b4776adc7c2519fcdf411 (table default.Table_46b7ffb5bf6f45f2be4d1b242f43bfc9 [id=0cd6320a10064e9a8556836dfdb2a1dc]) successfully deleted
I20251024 19:48:07.243790 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aebf76d4a66947ed9756643740418798" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:07.244563 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 7deb97be82964ea0b2fe9caa867a6460 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:07.244318 1444 leader_election.cc:343] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.251611 1775 log.cc:1199] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/f346a90a9efe442f82750336e3893deb
I20251024 19:48:07.252218 1775 ts_tablet_manager.cc:1950] T f346a90a9efe442f82750336e3893deb P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:07.253234 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 417) took 3366 ms. Trace:
I20251024 19:48:07.258242 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d76410afd7ba4c73836f4791ef739056" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:07.258493 1647 raft_consensus.cc:3060] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.261363 1768 tablet_bootstrap.cc:654] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.261693 2431 raft_consensus.cc:493] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.261955 2431 raft_consensus.cc:515] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.263911 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "452f77091adc4cf09de2231f2f6d845b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:07.264146 1648 raft_consensus.cc:2468] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:07.263036 2431 leader_election.cc:290] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.264638 1444 leader_election.cc:304] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:07.265224 2431 raft_consensus.cc:2804] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:07.265380 2431 raft_consensus.cc:493] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:07.265570 2431 raft_consensus.cc:3060] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.265784 1767 tablet_bootstrap.cc:492] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:07.266021 1767 ts_tablet_manager.cc:1403] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.287s user 0.011s sys 0.003s
I20251024 19:48:07.266925 1767 raft_consensus.cc:359] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.267220 1767 raft_consensus.cc:385] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.267341 1767 raft_consensus.cc:740] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.267676 1767 consensus_queue.cc:260] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.268617 1767 ts_tablet_manager.cc:1434] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:07.269001 1767 tablet_bootstrap.cc:492] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:07.268981 1189 master_service.cc:954] ReplaceTablet: received request to replace tablet 4193750331874d7ba9bbe6279c3a8e37 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:07.249927 1766 raft_consensus.cc:740] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.269634 2431 raft_consensus.cc:515] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.269701 1766 consensus_queue.cc:260] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.270845 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "452f77091adc4cf09de2231f2f6d845b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:07.271027 1650 raft_consensus.cc:3060] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.271054 2431 leader_election.cc:290] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.253350 1775 rpcz_store.cc:276] 1024 19:48:03.886441 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.892617 (+ 6176us) service_pool.cc:225] Handling call
1024 19:48:07.192562 (+3299945us) ts_tablet_manager.cc:1151] Deleting tablet f346a90a9efe442f82750336e3893deb
1024 19:48:07.192577 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.232099 (+ 39522us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.240410 (+ 8311us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.253173 (+ 12763us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3299564}
I20251024 19:48:07.273411 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 431) took 2929 ms. Trace:
I20251024 19:48:07.273837 1775 rpcz_store.cc:276] 1024 19:48:04.344037 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.344096 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:07.272435 (+2928339us) ts_tablet_manager.cc:1151] Deleting tablet d38154d3ffd34f4b8d7305b1c23a4fae
1024 19:48:07.272567 (+ 132us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.273146 (+ 579us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2927958}
I20251024 19:48:07.277271 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 432) took 2932 ms. Trace:
I20251024 19:48:07.278775 1650 raft_consensus.cc:2468] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:07.279279 1444 leader_election.cc:304] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:07.280104 1767 tablet_bootstrap.cc:654] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.278594 1775 rpcz_store.cc:276] 1024 19:48:04.344655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.344714 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:07.275771 (+2931057us) ts_tablet_manager.cc:1151] Deleting tablet d1d26e02ad2945b39cf27738ed5b7cf4
1024 19:48:07.275920 (+ 149us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.276639 (+ 719us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2930793}
I20251024 19:48:07.280730 2431 raft_consensus.cc:2804] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:07.281196 2431 raft_consensus.cc:697] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:07.282174 2431 consensus_queue.cc:237] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.283099 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 433) took 2937 ms. Trace:
I20251024 19:48:07.284015 1775 rpcz_store.cc:276] 1024 19:48:04.345142 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.345233 (+ 91us) service_pool.cc:225] Handling call
1024 19:48:07.281140 (+2935907us) ts_tablet_manager.cc:1151] Deleting tablet 4243f11bf2b34b30b10766c19e2c6409
1024 19:48:07.281413 (+ 273us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.282457 (+ 1044us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2935740}
I20251024 19:48:07.287441 1775 tablet_replica.cc:333] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:07.289215 1775 raft_consensus.cc:2243] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:07.291146 1766 ts_tablet_manager.cc:1434] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.056s user 0.001s sys 0.001s
I20251024 19:48:07.271790 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet f346a90a9efe442f82750336e3893deb (table default.Table_673f2f271b3e4f9d9c5f6802a3646f8f [id=b79808d531b740d7abc0e6fc2d9ba219]) successfully deleted
W20251024 19:48:07.291882 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet d38154d3ffd34f4b8d7305b1c23a4fae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d38154d3ffd34f4b8d7305b1c23a4fae already in progress: creating tablet
W20251024 19:48:07.292178 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet d1d26e02ad2945b39cf27738ed5b7cf4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1d26e02ad2945b39cf27738ed5b7cf4 already in progress: creating tablet
W20251024 19:48:07.292438 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4243f11bf2b34b30b10766c19e2c6409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4243f11bf2b34b30b10766c19e2c6409 already in progress: creating tablet
I20251024 19:48:07.293092 1766 tablet_bootstrap.cc:492] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:07.293143 2020 raft_consensus.cc:493] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.293848 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 7deb97be82964ea0b2fe9caa867a6460 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:07.295580 2020 raft_consensus.cc:515] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.296106 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6651e9c63f1d4f19b3d529bff417377b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:07.296234 2020 leader_election.cc:290] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:07.296639 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6651e9c63f1d4f19b3d529bff417377b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.297194 1188 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_147e888401aa40e5a60d97884b36cb98 [id=1acdc5d0bfb94e0faf47055b20d803b0] alter complete (version 1)
I20251024 19:48:07.297745 1766 tablet_bootstrap.cc:654] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:07.298410 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59956dad06064024ab171631ef6c1dca consensus state has no leader
I20251024 19:48:07.261504 1647 raft_consensus.cc:2468] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:07.298767 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_10657b3692ee4b7081e10b2ce7e87b07" } new_table_name: "default.Table_3834cd40435b4874a48fbec099663c9b" modify_external_catalogs: true
I20251024 19:48:07.300299 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5fd79a379e3b43aeb80de8af15bf4427" } modify_external_catalogs: true
I20251024 19:48:07.300482 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5fd79a379e3b43aeb80de8af15bf4427" } modify_external_catalogs: true
I20251024 19:48:07.301254 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_52ecaf891a2449d78bb8a354e7fb904b" } modify_external_catalogs: true
I20251024 19:48:07.301450 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_52ecaf891a2449d78bb8a354e7fb904b" } modify_external_catalogs: true
I20251024 19:48:07.301859 1766 tablet_bootstrap.cc:492] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:07.302044 1766 ts_tablet_manager.cc:1403] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20251024 19:48:07.302871 1775 pending_rounds.cc:70] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e: Trying to abort 1 pending ops.
I20251024 19:48:07.302932 1775 pending_rounds.cc:77] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429335742595072 op_type: NO_OP noop_request { }
I20251024 19:48:07.303609 1776 ts_tablet_manager.cc:1929] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.309928 1776 log.cc:1199] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/dc1b6f0a377945089c50012f74d38564
I20251024 19:48:07.310267 1444 leader_election.cc:304] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:07.310849 2020 raft_consensus.cc:2804] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:07.311005 2020 raft_consensus.cc:697] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:07.311815 2020 consensus_queue.cc:237] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.313817 1776 ts_tablet_manager.cc:1950] T dc1b6f0a377945089c50012f74d38564 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:07.316020 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b96fc2309274048bed7a35e4bf4bae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.316507 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet d76410afd7ba4c73836f4791ef739056 with delete_type TABLET_DATA_DELETED (replaced by tablet b153f75caebb4d869e77529c9c9bb50d) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.323526 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d76410afd7ba4c73836f4791ef739056" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:07.324530 1775 raft_consensus.cc:2889] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:07.324960 1775 raft_consensus.cc:2272] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.315752 1766 raft_consensus.cc:359] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:07.327747 1775 ts_tablet_manager.cc:1916] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:07.327960 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46e5e391a7674b5d8e1dd1a780fe36c8 consensus state has no leader
I20251024 19:48:07.328029 1766 raft_consensus.cc:385] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.330190 1766 raft_consensus.cc:740] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.330499 1766 consensus_queue.cc:260] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:07.332278 1766 ts_tablet_manager.cc:1434] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20251024 19:48:07.333030 2402 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:54174 (request call id 449) took 1104 ms. Trace:
I20251024 19:48:07.333866 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 476) took 1761 ms. Trace:
I20251024 19:48:07.334108 1766 tablet_bootstrap.cc:492] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:07.335114 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 2602335327fc489fa154056c5ef5138f from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:07.338002 1766 tablet_bootstrap.cc:654] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.336179 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad81d80697984ff1834a7d795e0f4b9b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:07.340189 1444 leader_election.cc:343] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.339587 1767 tablet_bootstrap.cc:492] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:07.339556 2321 raft_consensus.cc:493] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.340708 2321 raft_consensus.cc:515] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.341404 2321 leader_election.cc:290] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.342295 1767 ts_tablet_manager.cc:1403] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.000s
I20251024 19:48:07.343168 1767 raft_consensus.cc:359] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:07.343415 1767 raft_consensus.cc:385] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.343575 1767 raft_consensus.cc:740] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.343892 1767 consensus_queue.cc:260] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:07.344069 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a12e415590a4c71aab4ad7d30412c83 with delete_type TABLET_DATA_DELETED (replaced by tablet d6de2c21796f4bc4904f88b654c3baec) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:07.344305 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet fc7ccf1495924782b9b6ac1173bc73a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:07.345866 1767 ts_tablet_manager.cc:1434] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251024 19:48:07.346208 1767 tablet_bootstrap.cc:492] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:07.346627 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aebf76d4a66947ed9756643740418798" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:07.347462 1776 rpcz_store.cc:276] 1024 19:48:05.572728 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.573085 (+ 357us) service_pool.cc:225] Handling call
1024 19:48:07.239581 (+1666496us) ts_tablet_manager.cc:1151] Deleting tablet dc1b6f0a377945089c50012f74d38564
1024 19:48:07.239596 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.255903 (+ 16307us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.303417 (+ 47514us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.333819 (+ 30402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1666208}
I20251024 19:48:07.347922 1776 tablet_replica.cc:333] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:07.348387 1776 raft_consensus.cc:2243] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:07.348634 1776 raft_consensus.cc:2272] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.349874 2393 tablet.cc:1717] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532: Skipping requested alter to schema version 6, tablet already version 6
I20251024 19:48:07.352962 2439 tablet.cc:1717] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e: Skipping requested alter to schema version 6, tablet already version 6
W20251024 19:48:07.354200 1447 leader_election.cc:343] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.354449 1371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fffc42ddda14bd28800917ff2f0ced5" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:07.356264 1447 leader_election.cc:343] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.356364 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 0266cef4e2034303978d7456c47ea98c
I20251024 19:48:07.356818 1186 catalog_manager.cc:6719] ReplaceTablet: tablet 0266cef4e2034303978d7456c47ea98c deleted and replaced by tablet 64325d89db714554866e74922d103e34
I20251024 19:48:07.357170 1367 raft_consensus.cc:2468] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:07.357522 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet dc1b6f0a377945089c50012f74d38564 (table default.Table_52ec55c23c394294a68850ab6b33aabe [id=fb0e7cab20f44aaebdb09862bd193f41]) successfully deleted
I20251024 19:48:07.357964 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 0266cef4e2034303978d7456c47ea98c with delete_type TABLET_DATA_DELETED (replaced by tablet 64325d89db714554866e74922d103e34) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:07.358434 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 0266cef4e2034303978d7456c47ea98c with delete_type TABLET_DATA_DELETED (replaced by tablet 64325d89db714554866e74922d103e34) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.358875 1447 leader_election.cc:304] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:07.359279 2020 raft_consensus.cc:2804] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:07.359444 2020 raft_consensus.cc:493] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:07.359615 2020 raft_consensus.cc:3060] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.360220 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db17d9aff6542b6b94c09f872a4cceb" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:07.360803 1776 ts_tablet_manager.cc:1916] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.357038 1365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "452f77091adc4cf09de2231f2f6d845b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:07.362775 1447 leader_election.cc:343] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.362421 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad81d80697984ff1834a7d795e0f4b9b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:07.363608 1308 consensus_peers.cc:597] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5473699c1b647f8b86ee43fdb56bdfd. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:07.363628 1447 leader_election.cc:343] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.363751 1447 leader_election.cc:304] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:07.364187 2440 raft_consensus.cc:2749] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:07.364487 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 0266cef4e2034303978d7456c47ea98c with delete_type TABLET_DATA_DELETED (replaced by tablet 64325d89db714554866e74922d103e34) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:07.361943 1586 leader_election.cc:343] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: Tablet not RUNNING: STOPPED
I20251024 19:48:07.366374 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4193750331874d7ba9bbe6279c3a8e37
I20251024 19:48:07.363443 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "452f77091adc4cf09de2231f2f6d845b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:07.366909 1189 catalog_manager.cc:6719] ReplaceTablet: tablet 4193750331874d7ba9bbe6279c3a8e37 deleted and replaced by tablet ec2ffe32f2914ef59d5d77056fbb7d16
I20251024 19:48:07.367084 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db17d9aff6542b6b94c09f872a4cceb" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:07.367125 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 4193750331874d7ba9bbe6279c3a8e37 with delete_type TABLET_DATA_DELETED (replaced by tablet ec2ffe32f2914ef59d5d77056fbb7d16) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.362164 1767 tablet_bootstrap.cc:654] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:07.368237 1447 leader_election.cc:343] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.369220 2020 raft_consensus.cc:515] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:07.333164 2402 rpcz_store.cc:276] 1024 19:48:06.228786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.228849 (+ 63us) service_pool.cc:225] Handling call
1024 19:48:07.332988 (+1104139us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1024 19:48:06.264420 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1024 19:48:06.264481 (+ 61us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1024 19:48:06.264487 (+ 6us) alter_schema_op.cc:66] Acquired schema lock
1024 19:48:06.264492 (+ 5us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1024 19:48:06.264535 (+ 43us) alter_schema_op.cc:128] START. Timestamp: 1761335286264506
1024 19:48:06.264538 (+ 3us) op_driver.cc:348] REPLICATION: starting
1024 19:48:06.281169 (+ 16631us) log.cc:844] Serialized 173 byte log entry
1024 19:48:06.572041 (+290872us) op_driver.cc:464] REPLICATION: finished
1024 19:48:06.976326 (+404285us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1024 19:48:07.087654 (+111328us) log.cc:844] Serialized 5 byte log entry
1024 19:48:07.332074 (+244420us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.332390 (+ 316us) log.cc:844] Serialized 14 byte log entry
1024 19:48:07.332938 (+ 548us) alter_schema_op.cc:73] Released schema lock
1024 19:48:07.332944 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"spinlock_wait_cycles":22016,"child_traces":[["op",{"apply.queue_time_us":404097,"mutex_wait_us":50828,"prepare.queue_time_us":35162,"prepare.run_cpu_time_us":13941,"prepare.run_wall_time_us":193965,"raft.queue_time_us":51878,"raft.run_cpu_time_us":625,"raft.run_wall_time_us":763,"replication_time_us":305367,"thread_start_us":777601,"threads_started":3,"wal-append.queue_time_us":95740}]]}
I20251024 19:48:07.370532 2020 leader_election.cc:290] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:07.370644 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aebf76d4a66947ed9756643740418798" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:07.370712 1368 raft_consensus.cc:2393] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
I20251024 19:48:07.370849 1650 raft_consensus.cc:3060] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.372383 1766 tablet_bootstrap.cc:492] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:07.372534 1766 ts_tablet_manager.cc:1403] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20251024 19:48:07.373306 1766 raft_consensus.cc:359] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.373498 1766 raft_consensus.cc:385] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.374177 1650 raft_consensus.cc:2468] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:07.374629 1444 leader_election.cc:304] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:07.374617 1586 leader_election.cc:304] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:07.375058 2321 raft_consensus.cc:3060] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.375636 1192 catalog_manager.cc:5649] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:07.376037 1192 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 452f77091adc4cf09de2231f2f6d845b (table default.Table_0fa2426ca885458797e98bedc44dec04 [id=9a49120c5b654676a63cd3966eaf31a9]). Expected version 3 got 0
I20251024 19:48:07.376403 1192 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 7503b22d9e12481b9af96385b9d76e50 (table default.Table_1c5fc102a45e49948a8157aa0b5e6949 [id=9065c680f4da42b5a6e5ca335c8e2f3b]). Expected version 2 got 0
I20251024 19:48:07.378165 2321 raft_consensus.cc:2749] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:07.379137 1774 ts_tablet_manager.cc:1929] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.379294 1774 log.cc:1199] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/7c357a2a81064b3891afda6d252898b9
I20251024 19:48:07.380231 1775 ts_tablet_manager.cc:1929] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.380470 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 4193750331874d7ba9bbe6279c3a8e37 with delete_type TABLET_DATA_DELETED (replaced by tablet ec2ffe32f2914ef59d5d77056fbb7d16) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:07.381634 1774 ts_tablet_manager.cc:1950] T 7c357a2a81064b3891afda6d252898b9 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:07.382879 1775 log.cc:1199] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/4db17d9aff6542b6b94c09f872a4cceb
I20251024 19:48:07.382718 1191 catalog_manager.cc:5649] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:07.382612 2020 raft_consensus.cc:2804] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:07.383237 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet ad055dc3387b414bb55d09b9683ba17e (table default.Table_cde3670bf0d04dc0b2049193bcee10d4 [id=c3de6937e93a48dcb88cabee59a09d32]). Expected version 1 got 0
I20251024 19:48:07.383913 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 376) took 3682 ms. Trace:
I20251024 19:48:07.383945 2020 raft_consensus.cc:697] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:07.384328 1774 rpcz_store.cc:276] 1024 19:48:03.699992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.700242 (+ 250us) service_pool.cc:225] Handling call
1024 19:48:06.921043 (+3220801us) ts_tablet_manager.cc:1151] Deleting tablet 7c357a2a81064b3891afda6d252898b9
1024 19:48:06.921050 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.168048 (+246998us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.373561 (+205513us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.382273 (+ 8712us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3220555}
I20251024 19:48:07.384289 2020 consensus_queue.cc:237] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:07.384786 1774 tablet_replica.cc:333] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:07.385063 1774 raft_consensus.cc:2243] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:07.385432 1774 raft_consensus.cc:2272] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.386191 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 4193750331874d7ba9bbe6279c3a8e37 with delete_type TABLET_DATA_DELETED (replaced by tablet ec2ffe32f2914ef59d5d77056fbb7d16) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:07.386445 1776 ts_tablet_manager.cc:1929] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.381237 1766 raft_consensus.cc:740] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.387161 1766 consensus_queue.cc:260] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.388630 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dedd7e29ac324761b962fbd47c7c25e9" } modify_external_catalogs: true
I20251024 19:48:07.388828 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dedd7e29ac324761b962fbd47c7c25e9" } modify_external_catalogs: true
I20251024 19:48:07.389734 1766 ts_tablet_manager.cc:1434] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20251024 19:48:07.389715 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aebf76d4a66947ed9756643740418798" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:07.390069 1766 tablet_bootstrap.cc:492] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:07.390043 1368 raft_consensus.cc:3060] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.390255 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9987a00cca5d4a8f873241da5a36b46a" } new_table_name: "default.Table_d5fdf824dcd943138d9d3ab80f5bee04" modify_external_catalogs: true
I20251024 19:48:07.391602 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_14c1f0af1332491a8ad3d0a6252f840d" } modify_external_catalogs: true
I20251024 19:48:07.387239 1767 tablet_bootstrap.cc:492] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:07.388811 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 7e0dc7d4fed5448cb616af29e405e766 (DEFAULT_TABLE table=default.Table_d601fa3e84af4fdc8768772e2283a86f [id=f6226be8495d421bb26960fbed1fd190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.391769 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_14c1f0af1332491a8ad3d0a6252f840d" } modify_external_catalogs: true
I20251024 19:48:07.392122 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e0dc7d4fed5448cb616af29e405e766. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.394055 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 7c357a2a81064b3891afda6d252898b9 (table default.Table_c570270b7d7b4f5091c34e0550dd46ef [id=ed879934f5754b92b8cff234f3f66333]) successfully deleted
I20251024 19:48:07.395900 1767 ts_tablet_manager.cc:1403] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20251024 19:48:07.396083 1774 ts_tablet_manager.cc:1916] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.396790 1767 raft_consensus.cc:359] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.397800 1767 raft_consensus.cc:385] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.398654 1767 raft_consensus.cc:740] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.399075 1767 consensus_queue.cc:260] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.401090 1767 ts_tablet_manager.cc:1434] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:48:07.401614 1767 tablet_bootstrap.cc:492] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:07.403502 1767 tablet_bootstrap.cc:654] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.404177 1766 tablet_bootstrap.cc:654] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.405345 1775 ts_tablet_manager.cc:1950] T 4db17d9aff6542b6b94c09f872a4cceb P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:07.406499 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 434) took 3060 ms. Trace:
I20251024 19:48:07.406695 1775 rpcz_store.cc:276] 1024 19:48:04.372613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.390106 (+ 17493us) service_pool.cc:225] Handling call
1024 19:48:07.286632 (+2896526us) ts_tablet_manager.cc:1151] Deleting tablet 4db17d9aff6542b6b94c09f872a4cceb
1024 19:48:07.286811 (+ 179us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.351449 (+ 64638us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.380083 (+ 28634us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.406409 (+ 26326us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2872342}
I20251024 19:48:07.407409 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 4db17d9aff6542b6b94c09f872a4cceb (table default.Table_ef3fdce598e74165bddac418dc92c20e [id=e4ee1d33b7e24827aa24950ea40465e4]) successfully deleted
I20251024 19:48:07.407539 1368 raft_consensus.cc:2468] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:07.409055 1776 log.cc:1199] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/030e690ce4df4818a6450ae293b0ffbd
I20251024 19:48:07.409667 1776 ts_tablet_manager.cc:1950] T 030e690ce4df4818a6450ae293b0ffbd P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:07.409677 1775 tablet_replica.cc:333] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:07.409866 1775 raft_consensus.cc:2243] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:07.409772 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0116d95669da49668f8aa797d948aae3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:07.410053 1775 raft_consensus.cc:2272] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:07.410356 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0116d95669da49668f8aa797d948aae3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.412595 1775 ts_tablet_manager.cc:1916] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.414340 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7712392f200842c4b728e019d6a3cfb0
I20251024 19:48:07.414445 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 477) took 1841 ms. Trace:
I20251024 19:48:07.414530 1776 rpcz_store.cc:276] 1024 19:48:05.573033 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.573669 (+ 636us) service_pool.cc:225] Handling call
1024 19:48:07.347862 (+1774193us) ts_tablet_manager.cc:1151] Deleting tablet 030e690ce4df4818a6450ae293b0ffbd
1024 19:48:07.347872 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.371316 (+ 23444us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.386318 (+ 15002us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.414388 (+ 28070us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1773886}
I20251024 19:48:07.415043 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 478) took 1795 ms. Trace:
I20251024 19:48:07.415134 1776 rpcz_store.cc:276] 1024 19:48:05.619829 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.620698 (+ 869us) service_pool.cc:225] Handling call
1024 19:48:07.414834 (+1794136us) ts_tablet_manager.cc:1151] Deleting tablet 3fd8aa1d9aca429dbfe46f472ebbf53b
1024 19:48:07.414844 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.414984 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1793592}
I20251024 19:48:07.415198 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9f30a26963bc4d0aa33b95bb45032870"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:07.415433 1776 tablet_replica.cc:333] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:07.415617 1776 raft_consensus.cc:2243] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
W20251024 19:48:07.415787 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f30a26963bc4d0aa33b95bb45032870 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.415928 1776 raft_consensus.cc:2272] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.415138 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 7712392f200842c4b728e019d6a3cfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.416417 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 7712392f200842c4b728e019d6a3cfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:07.416949 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet d4b3a50db1714eb38cc0ea03f612fc95
I20251024 19:48:07.417155 2438 tablet.cc:1717] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97: Skipping requested alter to schema version 6, tablet already version 6
I20251024 19:48:07.417631 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_631ebe0a47914d38854b5d13f0ea4af4" } modify_external_catalogs: true
I20251024 19:48:07.417791 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_631ebe0a47914d38854b5d13f0ea4af4" } modify_external_catalogs: true
I20251024 19:48:07.418817 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet d4b3a50db1714eb38cc0ea03f612fc95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:07.419493 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_13f216b61d134e4cba15849361a28394" } new_table_name: "default.Table_2021df09852e4dd09dbf7b36a0a3ef5b" modify_external_catalogs: true
I20251024 19:48:07.421092 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b519454f788a4f59bada834ba04bc981"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:07.421698 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b519454f788a4f59bada834ba04bc981 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.422909 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 030e690ce4df4818a6450ae293b0ffbd (table default.Table_4db7a2da5af4481db5b546d201dfd278 [id=2ac38aaa9204452787b18173393bb949]) successfully deleted
W20251024 19:48:07.423210 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fd8aa1d9aca429dbfe46f472ebbf53b already in progress: creating tablet
I20251024 19:48:07.424597 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c3453ee396f645c79eb00b1777d54f12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:07.424734 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet d4b3a50db1714eb38cc0ea03f612fc95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:07.424924 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 7712392f200842c4b728e019d6a3cfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:07.425261 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3453ee396f645c79eb00b1777d54f12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.429714 2321 raft_consensus.cc:493] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.429930 2321 raft_consensus.cc:515] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.430680 2321 leader_election.cc:290] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.430841 1776 ts_tablet_manager.cc:1916] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.431088 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ce8c179e98944df0b510a0e8daf3920f
I20251024 19:48:07.432291 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet ce8c179e98944df0b510a0e8daf3920f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:07.437677 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet ce8c179e98944df0b510a0e8daf3920f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.437887 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet d4b3a50db1714eb38cc0ea03f612fc95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.439144 1767 tablet_bootstrap.cc:492] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:07.439078 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3b7821f2ba4a40a6a1d6c6ae777feb70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:07.439308 1767 ts_tablet_manager.cc:1403] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.004s
I20251024 19:48:07.439586 1775 ts_tablet_manager.cc:1929] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251024 19:48:07.439764 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b7821f2ba4a40a6a1d6c6ae777feb70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.440696 1767 raft_consensus.cc:359] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.440959 1767 raft_consensus.cc:385] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.441190 1767 raft_consensus.cc:740] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.440516 1766 tablet_bootstrap.cc:492] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:07.441787 1766 ts_tablet_manager.cc:1403] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.002s
I20251024 19:48:07.441802 1767 consensus_queue.cc:260] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.442323 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet ce8c179e98944df0b510a0e8daf3920f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:07.442634 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a12e415590a4c71aab4ad7d30412c83" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:07.442682 1766 raft_consensus.cc:359] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.442845 1766 raft_consensus.cc:385] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.442934 1509 raft_consensus.cc:2393] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 43bec2f03fc849099e254f6335e9d532 in this term.
I20251024 19:48:07.442929 1766 raft_consensus.cc:740] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.443603 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a12e415590a4c71aab4ad7d30412c83" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:07.444206 1586 leader_election.cc:304] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:07.445063 2321 raft_consensus.cc:3060] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.447935 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fd7726c532145c6b44ea9a37544510d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.448050 2321 raft_consensus.cc:2749] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:07.448992 1767 ts_tablet_manager.cc:1434] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20251024 19:48:07.450290 1766 consensus_queue.cc:260] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:07.453687 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e747880c4e1425ba749201e624a083e has no consensus state
I20251024 19:48:07.455988 1767 tablet_bootstrap.cc:492] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:07.456454 1186 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 452f77091adc4cf09de2231f2f6d845b (table default.Table_0fa2426ca885458797e98bedc44dec04 [id=9a49120c5b654676a63cd3966eaf31a9]). Expected version 3 got 0
I20251024 19:48:07.456806 1766 ts_tablet_manager.cc:1434] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251024 19:48:07.462550 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 4db17d9aff6542b6b94c09f872a4cceb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:07.462797 1766 tablet_bootstrap.cc:492] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:07.463208 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet a76d61c3464b41519d7657796bd853ff with delete_type TABLET_DATA_DELETED (replaced by tablet b240965605904533bd2af99f61aaef26) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:07.463459 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet aebf76d4a66947ed9756643740418798 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:07.464017 1767 tablet_bootstrap.cc:654] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.464895 1766 tablet_bootstrap.cc:654] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.466660 1775 log.cc:1199] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/ad81d80697984ff1834a7d795e0f4b9b
I20251024 19:48:07.467128 1775 ts_tablet_manager.cc:1950] T ad81d80697984ff1834a7d795e0f4b9b P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:07.467753 1186 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 037c8cc1e53447109fafc047c02f196f (table default.Table_e87be853efba49aeb50e71270beac600 [id=5bc052b3e0f04699b4b391bcec33617d]). Expected version 1 got 0
I20251024 19:48:07.472991 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 435) took 3082 ms. Trace:
I20251024 19:48:07.473196 1775 rpcz_store.cc:276] 1024 19:48:04.390653 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.390707 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:07.407881 (+3017174us) ts_tablet_manager.cc:1151] Deleting tablet ad81d80697984ff1834a7d795e0f4b9b
1024 19:48:07.407891 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.436178 (+ 28287us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.439437 (+ 3259us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.472944 (+ 33507us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2993410}
I20251024 19:48:07.474678 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 430) took 3130 ms. Trace:
I20251024 19:48:07.474858 1775 rpcz_store.cc:276] 1024 19:48:04.343908 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.425893 (+ 81985us) service_pool.cc:225] Handling call
1024 19:48:07.474511 (+3048618us) ts_tablet_manager.cc:1151] Deleting tablet af43ffe097e14c4faffed95e207d0893
1024 19:48:07.474521 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.474639 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3048308}
I20251024 19:48:07.475498 1776 ts_tablet_manager.cc:1929] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.476042 1775 tablet_replica.cc:333] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:07.476387 1775 raft_consensus.cc:2243] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:07.476766 1775 pending_rounds.cc:70] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e: Trying to abort 1 pending ops.
I20251024 19:48:07.476856 1775 pending_rounds.cc:77] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429327344504832 op_type: NO_OP noop_request { }
I20251024 19:48:07.476982 1775 raft_consensus.cc:2889] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:07.477123 1775 raft_consensus.cc:2272] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.477730 1775 ts_tablet_manager.cc:1916] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.478669 2438 tablet.cc:1722] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:07.478816 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 672b571081ea4cd78028b6a25625103c (DEFAULT_TABLE table=default.Table_3c5c40909920463ab35ba50c7a03bd36 [id=bc1fce97c45b48608016bac2f76b18c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.479270 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 672b571081ea4cd78028b6a25625103c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.469092 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 94b35252e63f4a8ba305cc029ddb5ae8 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:07.479702 2431 raft_consensus.cc:493] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.479843 2431 raft_consensus.cc:515] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.480296 2439 tablet.cc:1722] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:07.480671 2431 leader_election.cc:290] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:07.476431 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet af43ffe097e14c4faffed95e207d0893 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af43ffe097e14c4faffed95e207d0893 already in progress: creating tablet
I20251024 19:48:07.482095 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b96fc2309274048bed7a35e4bf4bae2" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:07.482211 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 7e0dc7d4fed5448cb616af29e405e766 (DEFAULT_TABLE table=default.Table_d601fa3e84af4fdc8768772e2283a86f [id=f6226be8495d421bb26960fbed1fd190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.482757 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e0dc7d4fed5448cb616af29e405e766. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:07.482729 1444 leader_election.cc:343] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.487489 2431 raft_consensus.cc:493] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.487782 2431 raft_consensus.cc:515] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.489271 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fd7726c532145c6b44ea9a37544510d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:07.490110 1647 raft_consensus.cc:2468] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:07.490756 1444 leader_election.cc:304] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:07.491055 1776 log.cc:1199] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/44ee03ede02c4bdab706fde910978332
I20251024 19:48:07.491214 2440 raft_consensus.cc:2804] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:07.491452 2440 raft_consensus.cc:493] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:07.491621 2440 raft_consensus.cc:3060] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.491684 1776 ts_tablet_manager.cc:1950] T 44ee03ede02c4bdab706fde910978332 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:07.492537 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 480) took 1872 ms. Trace:
I20251024 19:48:07.492714 1776 rpcz_store.cc:276] 1024 19:48:05.620281 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.621645 (+ 1364us) service_pool.cc:225] Handling call
1024 19:48:07.415351 (+1793706us) ts_tablet_manager.cc:1151] Deleting tablet 44ee03ede02c4bdab706fde910978332
1024 19:48:07.415360 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.459880 (+ 44520us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.475260 (+ 15380us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.492496 (+ 17236us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1792836}
I20251024 19:48:07.494272 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4cac515c0a00403fb680454c1721be0a
I20251024 19:48:07.495682 2440 raft_consensus.cc:515] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.497934 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 44ee03ede02c4bdab706fde910978332 (table default.Table_5f1d0b12b2034f5185f21c87157cf051 [id=46806f1d4f434926bc23214f0e3605d8]) successfully deleted
I20251024 19:48:07.498348 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 4cac515c0a00403fb680454c1721be0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:07.499346 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fd7726c532145c6b44ea9a37544510d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:07.499711 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 481) took 1878 ms. Trace:
I20251024 19:48:07.499730 1650 raft_consensus.cc:3060] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.500885 1189 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet a3259823b622461fa823841925b3d203 (table default.Table_ea0d3589c13a4ebeb950ddc737a9948b [id=350b37bf42194fcfbb660bffe83e47c4]). Expected version 4 got 0
I20251024 19:48:07.501194 1767 tablet_bootstrap.cc:492] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:07.501600 1776 rpcz_store.cc:276] 1024 19:48:05.620614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.621901 (+ 1287us) service_pool.cc:225] Handling call
1024 19:48:07.497761 (+1875860us) ts_tablet_manager.cc:1151] Deleting tablet 505c30f0b2b749ce918b578903db3757
1024 19:48:07.497773 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.498706 (+ 933us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1874940}
I20251024 19:48:07.502177 1767 ts_tablet_manager.cc:1403] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.005s
W20251024 19:48:07.503705 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 505c30f0b2b749ce918b578903db3757 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 505c30f0b2b749ce918b578903db3757 already in progress: creating tablet
I20251024 19:48:07.503738 1767 raft_consensus.cc:359] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.504099 1767 raft_consensus.cc:385] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.504060 1189 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet ce0ef4443e574846a9f48fef3b893352 (table default.Table_f203b363de4c47fca0a0599140d18279 [id=ec57efb4d37e44f1b68045f58bc11c45]). Expected version 1 got 0
I20251024 19:48:07.504261 1767 raft_consensus.cc:740] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.504617 1767 consensus_queue.cc:260] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.505303 1650 raft_consensus.cc:2468] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:07.506070 2440 leader_election.cc:290] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.506760 1444 leader_election.cc:304] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:07.507287 2440 raft_consensus.cc:2804] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:07.507469 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 7e0dc7d4fed5448cb616af29e405e766 (DEFAULT_TABLE table=default.Table_d601fa3e84af4fdc8768772e2283a86f [id=f6226be8495d421bb26960fbed1fd190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.507793 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 73ffe342f541447d8519ee24bc98a0f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 505e4fb0755a4caaa462e5f5154bfee2) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.508159 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e0dc7d4fed5448cb616af29e405e766. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.508178 2440 raft_consensus.cc:697] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:07.508821 1767 ts_tablet_manager.cc:1434] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20251024 19:48:07.508817 2440 consensus_queue.cc:237] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.509706 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 4cac515c0a00403fb680454c1721be0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:07.510044 1776 tablet_replica.cc:333] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:07.510141 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b96fc2309274048bed7a35e4bf4bae2" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:07.510511 1776 raft_consensus.cc:2243] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:07.510775 2431 leader_election.cc:290] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.511019 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 4cac515c0a00403fb680454c1721be0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.511348 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b04513e51b4647aebeee3f6e97e6c8e6" } new_table_name: "default.Table_83b4409228694c939a4a734c3b6bdd76" modify_external_catalogs: true
I20251024 19:48:07.511511 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fd7726c532145c6b44ea9a37544510d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:07.517834 1776 raft_consensus.cc:2272] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.518246 1767 tablet_bootstrap.cc:492] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:07.518282 1766 tablet_bootstrap.cc:492] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:07.518447 1766 ts_tablet_manager.cc:1403] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.000s
W20251024 19:48:07.511317 1447 leader_election.cc:343] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.518828 1447 leader_election.cc:304] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:07.519098 2431 raft_consensus.cc:2749] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:07.519284 2020 raft_consensus.cc:493] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.519449 2020 raft_consensus.cc:515] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.519958 1767 tablet_bootstrap.cc:654] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.520139 2020 leader_election.cc:290] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:07.520989 1447 leader_election.cc:343] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.521683 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce0ef4443e574846a9f48fef3b893352" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:07.521906 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fd7726c532145c6b44ea9a37544510d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:07.521967 1647 raft_consensus.cc:2468] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:07.522518 1444 leader_election.cc:304] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:07.522825 2020 raft_consensus.cc:2804] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:07.522912 2020 raft_consensus.cc:493] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:07.523010 2020 raft_consensus.cc:3060] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.523000 1776 ts_tablet_manager.cc:1916] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.525075 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce0ef4443e574846a9f48fef3b893352" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:07.526038 2402 tablet.cc:1722] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:07.526258 2020 raft_consensus.cc:515] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.526499 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 95f62010a24f432c84c89345f9b02b2f (DEFAULT_TABLE table=default.Table_05b0fc0694f240dda1d013cc7e547b82 [id=631df7faaad443ac93d6c914177f9453]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.526963 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f62010a24f432c84c89345f9b02b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.528895 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 55aa786b9f6847baa265802cc28b2b80 (DEFAULT_TABLE table=default.Table_370e897b37284a38948769e22c2d22f3 [id=d300307b4cae4f0cbbf98b5a026e5ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.529273 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55aa786b9f6847baa265802cc28b2b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.530215 2020 leader_election.cc:290] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.530700 2372 consensus_queue.cc:1048] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:07.538066 1447 leader_election.cc:343] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:07.538620 1447 leader_election.cc:343] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.539357 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 672b571081ea4cd78028b6a25625103c (DEFAULT_TABLE table=default.Table_3c5c40909920463ab35ba50c7a03bd36 [id=bc1fce97c45b48608016bac2f76b18c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.539844 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 672b571081ea4cd78028b6a25625103c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.540082 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 95f62010a24f432c84c89345f9b02b2f (DEFAULT_TABLE table=default.Table_05b0fc0694f240dda1d013cc7e547b82 [id=631df7faaad443ac93d6c914177f9453]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.540593 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f62010a24f432c84c89345f9b02b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.544360 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 55aa786b9f6847baa265802cc28b2b80 (DEFAULT_TABLE table=default.Table_370e897b37284a38948769e22c2d22f3 [id=d300307b4cae4f0cbbf98b5a026e5ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.546137 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 95f62010a24f432c84c89345f9b02b2f (DEFAULT_TABLE table=default.Table_05b0fc0694f240dda1d013cc7e547b82 [id=631df7faaad443ac93d6c914177f9453]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.546571 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f62010a24f432c84c89345f9b02b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.547188 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55aa786b9f6847baa265802cc28b2b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.547911 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 672b571081ea4cd78028b6a25625103c (DEFAULT_TABLE table=default.Table_3c5c40909920463ab35ba50c7a03bd36 [id=bc1fce97c45b48608016bac2f76b18c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.548301 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 672b571081ea4cd78028b6a25625103c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.553992 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 55aa786b9f6847baa265802cc28b2b80 (DEFAULT_TABLE table=default.Table_370e897b37284a38948769e22c2d22f3 [id=d300307b4cae4f0cbbf98b5a026e5ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.554450 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55aa786b9f6847baa265802cc28b2b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.558844 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce0ef4443e574846a9f48fef3b893352" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:07.559201 1650 raft_consensus.cc:3060] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.563336 1650 raft_consensus.cc:2468] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:07.567893 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b20ffadae0f24d4e996d915faeef9ddb" } new_table_name: "default.Table_1cf2ee7da8ea49c399e3e7cbb4592c8f" modify_external_catalogs: true
I20251024 19:48:07.567960 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce0ef4443e574846a9f48fef3b893352" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:07.568732 1447 leader_election.cc:343] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.576982 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b2f56bfc63f44d61b5edaa6bf59b7009"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:07.577666 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2f56bfc63f44d61b5edaa6bf59b7009 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.578963 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 1295c99836d14a1cb0002e6c6aaee1eb
I20251024 19:48:07.579670 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ad7f12b3760d494db551de925da842df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:07.580219 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad7f12b3760d494db551de925da842df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:07.581465 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 292c31e7d4ce411bb336a1405ca43ba5 consensus state has no leader
I20251024 19:48:07.581828 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8d2ad543f7a04e9e983cdaa997e94073"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:07.582132 2402 tablet.cc:1722] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251024 19:48:07.582351 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d2ad543f7a04e9e983cdaa997e94073 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.582971 1766 raft_consensus.cc:359] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.583159 1766 raft_consensus.cc:385] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.583248 1766 raft_consensus.cc:740] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.563556 1776 ts_tablet_manager.cc:1929] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.584393 1776 log.cc:1199] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/e3b4e85f741849b2a6a3e8574396ff88
I20251024 19:48:07.584633 1444 leader_election.cc:304] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:07.584928 1776 ts_tablet_manager.cc:1950] T e3b4e85f741849b2a6a3e8574396ff88 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:07.585139 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 1295c99836d14a1cb0002e6c6aaee1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:07.585350 2440 raft_consensus.cc:2804] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:07.585625 2440 raft_consensus.cc:697] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:07.585695 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 59eca6bb1dd84c67a7e53131abaa937a (DEFAULT_TABLE table=default.Table_accb35a3be634901b43dbbdf579098cd [id=4acdceff031e469f9a6f8211866f16df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.585865 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 479) took 1965 ms. Trace:
I20251024 19:48:07.585898 2440 consensus_queue.cc:237] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.585964 1776 rpcz_store.cc:276] 1024 19:48:05.620015 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.621290 (+ 1275us) service_pool.cc:225] Handling call
1024 19:48:07.509973 (+1888683us) ts_tablet_manager.cc:1151] Deleting tablet e3b4e85f741849b2a6a3e8574396ff88
1024 19:48:07.509984 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.554350 (+ 44366us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.557550 (+ 3200us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.585803 (+ 28253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1886757}
I20251024 19:48:07.586133 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59eca6bb1dd84c67a7e53131abaa937a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.586077 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_05b0fc0694f240dda1d013cc7e547b82" } new_table_name: "default.Table_cd17bd14f2964c739da271f669e63cd7" modify_external_catalogs: true
I20251024 19:48:07.586480 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 486) took 1950 ms. Trace:
I20251024 19:48:07.586566 1776 rpcz_store.cc:276] 1024 19:48:05.636210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.636282 (+ 72us) service_pool.cc:225] Handling call
1024 19:48:07.586268 (+1949986us) ts_tablet_manager.cc:1151] Deleting tablet 7c880a4af94d4041b8c7cf2066002ee3
1024 19:48:07.586281 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.586428 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1949715}
I20251024 19:48:07.586926 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 487) took 1928 ms. Trace:
I20251024 19:48:07.587009 1776 rpcz_store.cc:276] 1024 19:48:05.658317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.658378 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:07.586777 (+1928399us) ts_tablet_manager.cc:1151] Deleting tablet 7da1601a87524ef1a8b5069ba6823843
1024 19:48:07.586786 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.586874 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1927949}
I20251024 19:48:07.587334 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 489) took 1815 ms. Trace:
W20251024 19:48:07.587410 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 7e747880c4e1425ba749201e624a083e (table default.Table_d5fdf824dcd943138d9d3ab80f5bee04 [id=619e60f84da24d899ab2f69923f459b1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7e747880c4e1425ba749201e624a083e has no consensus state
I20251024 19:48:07.587442 1776 rpcz_store.cc:276] 1024 19:48:05.772071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.772117 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:07.587197 (+1815080us) ts_tablet_manager.cc:1151] Deleting tablet 505e4fb0755a4caaa462e5f5154bfee2
1024 19:48:07.587206 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.587285 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1814784}
I20251024 19:48:07.587811 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 498) took 1210 ms. Trace:
I20251024 19:48:07.587908 1776 rpcz_store.cc:276] 1024 19:48:06.377254 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.377322 (+ 68us) service_pool.cc:225] Handling call
1024 19:48:07.587663 (+1210341us) ts_tablet_manager.cc:1151] Deleting tablet 780442b310e6409498e6bbe4256280a4
1024 19:48:07.587673 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.587751 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1210066}
I20251024 19:48:07.587963 1766 consensus_queue.cc:260] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.588210 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 1295c99836d14a1cb0002e6c6aaee1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:07.588287 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 499) took 1210 ms. Trace:
I20251024 19:48:07.588359 1776 rpcz_store.cc:276] 1024 19:48:06.377891 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.377946 (+ 55us) service_pool.cc:225] Handling call
1024 19:48:07.588123 (+1210177us) ts_tablet_manager.cc:1151] Deleting tablet 65424b00ab9c454f9584048caba83e6a
1024 19:48:07.588131 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.588235 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1209929}
I20251024 19:48:07.588481 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet e3b4e85f741849b2a6a3e8574396ff88 (table default.Table_b905a1285caa407e9fb08e81639c9bc8 [id=1da47e6010f548d1b3d8c538150a8a8e]) successfully deleted
I20251024 19:48:07.588658 1766 ts_tablet_manager.cc:1434] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.070s user 0.002s sys 0.000s
I20251024 19:48:07.588740 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 500) took 1210 ms. Trace:
W20251024 19:48:07.588735 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7c880a4af94d4041b8c7cf2066002ee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c880a4af94d4041b8c7cf2066002ee3 already in progress: creating tablet
I20251024 19:48:07.588819 1776 rpcz_store.cc:276] 1024 19:48:06.378400 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.378453 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:07.588609 (+1210156us) ts_tablet_manager.cc:1151] Deleting tablet 3324755a5a184a609fae5f7a25d58e4c
1024 19:48:07.588617 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.588690 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1209943}
I20251024 19:48:07.588963 1766 tablet_bootstrap.cc:492] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:07.588981 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7da1601a87524ef1a8b5069ba6823843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7da1601a87524ef1a8b5069ba6823843 already in progress: creating tablet
I20251024 19:48:07.589165 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 501) took 1210 ms. Trace:
W20251024 19:48:07.589205 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 505e4fb0755a4caaa462e5f5154bfee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 505e4fb0755a4caaa462e5f5154bfee2 already in progress: creating tablet
I20251024 19:48:07.589262 1776 rpcz_store.cc:276] 1024 19:48:06.378958 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.379011 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:07.589029 (+1210018us) ts_tablet_manager.cc:1151] Deleting tablet 7eb62791100f477784ab315923b6ceb6
1024 19:48:07.589037 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.589114 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1209772}
I20251024 19:48:07.567178 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 57103d6696d64848adee2fe976a22c93 (DEFAULT_TABLE table=default.Table_407e3b1465004f64bcabe7e2da5d4935 [id=c361c7f277c74818b31261921c102d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.590030 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57103d6696d64848adee2fe976a22c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.590366 1774 ts_tablet_manager.cc:1929] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.590523 1774 log.cc:1199] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/857eed60ba2a4b309e86d88a0f093762
W20251024 19:48:07.590554 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 780442b310e6409498e6bbe4256280a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 780442b310e6409498e6bbe4256280a4 already in progress: creating tablet
W20251024 19:48:07.590809 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 65424b00ab9c454f9584048caba83e6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65424b00ab9c454f9584048caba83e6a already in progress: creating tablet
I20251024 19:48:07.590958 1766 tablet_bootstrap.cc:654] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.591004 1774 ts_tablet_manager.cc:1950] T 857eed60ba2a4b309e86d88a0f093762 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:48:07.591094 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3324755a5a184a609fae5f7a25d58e4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3324755a5a184a609fae5f7a25d58e4c already in progress: creating tablet
W20251024 19:48:07.591382 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7eb62791100f477784ab315923b6ceb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7eb62791100f477784ab315923b6ceb6 already in progress: creating tablet
I20251024 19:48:07.591878 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 378) took 3844 ms. Trace:
I20251024 19:48:07.591975 1774 rpcz_store.cc:276] 1024 19:48:03.747190 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.747651 (+ 461us) service_pool.cc:225] Handling call
1024 19:48:07.384706 (+3637055us) ts_tablet_manager.cc:1151] Deleting tablet 857eed60ba2a4b309e86d88a0f093762
1024 19:48:07.384717 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.411425 (+ 26708us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.581049 (+169624us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.591818 (+ 10769us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3635709}
I20251024 19:48:07.591965 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_afee33bb95144492a2c62b19a101c0e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:07.592442 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 377) took 3851 ms. Trace:
W20251024 19:48:07.592706 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afee33bb95144492a2c62b19a101c0e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.592674 1774 rpcz_store.cc:276] 1024 19:48:03.741321 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.741528 (+ 207us) service_pool.cc:225] Handling call
1024 19:48:07.592281 (+3850753us) ts_tablet_manager.cc:1151] Deleting tablet 45cff9bcc4924e0081518c284f074940
1024 19:48:07.592293 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.592390 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3841007}
I20251024 19:48:07.593092 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 379) took 3815 ms. Trace:
I20251024 19:48:07.593173 1774 rpcz_store.cc:276] 1024 19:48:03.778047 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.778691 (+ 644us) service_pool.cc:225] Handling call
1024 19:48:07.592940 (+3814249us) ts_tablet_manager.cc:1151] Deleting tablet 2602335327fc489fa154056c5ef5138f
1024 19:48:07.592950 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.593041 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3813981}
I20251024 19:48:07.593452 1774 tablet_replica.cc:333] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:07.567339 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 59eca6bb1dd84c67a7e53131abaa937a (DEFAULT_TABLE table=default.Table_accb35a3be634901b43dbbdf579098cd [id=4acdceff031e469f9a6f8211866f16df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.594352 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59eca6bb1dd84c67a7e53131abaa937a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.594803 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 1295c99836d14a1cb0002e6c6aaee1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:07.594300 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 59eca6bb1dd84c67a7e53131abaa937a (DEFAULT_TABLE table=default.Table_accb35a3be634901b43dbbdf579098cd [id=4acdceff031e469f9a6f8211866f16df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.597268 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b25ae72792dd4a14adeb24e822c8b538"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:07.578596 1767 tablet_bootstrap.cc:492] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:07.597743 1767 ts_tablet_manager.cc:1403] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.007s
W20251024 19:48:07.598028 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b25ae72792dd4a14adeb24e822c8b538 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.598737 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59eca6bb1dd84c67a7e53131abaa937a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.598644 1767 raft_consensus.cc:359] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.598825 1767 raft_consensus.cc:385] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.598915 1767 raft_consensus.cc:740] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.599149 1767 consensus_queue.cc:260] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.599709 1767 ts_tablet_manager.cc:1434] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:07.600025 1767 tablet_bootstrap.cc:492] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:07.609148 1767 tablet_bootstrap.cc:654] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.610767 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 504) took 1151 ms. Trace:
I20251024 19:48:07.610886 1776 rpcz_store.cc:276] 1024 19:48:06.438309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.439244 (+ 935us) service_pool.cc:225] Handling call
1024 19:48:07.589473 (+1150229us) ts_tablet_manager.cc:1151] Deleting tablet 83b91a5d764449d5b82ef6d1ee0360f5
1024 19:48:07.589484 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.589563 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1150014}
I20251024 19:48:07.611455 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 505) took 1172 ms. Trace:
I20251024 19:48:07.611552 1776 rpcz_store.cc:276] 1024 19:48:06.438454 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.439494 (+ 1040us) service_pool.cc:225] Handling call
1024 19:48:07.611201 (+1171707us) ts_tablet_manager.cc:1151] Deleting tablet 94b35252e63f4a8ba305cc029ddb5ae8
1024 19:48:07.611212 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.611363 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171547}
I20251024 19:48:07.611979 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 509) took 1051 ms. Trace:
I20251024 19:48:07.611940 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 14965e5ea7ae43f2b80986c58a2e9f2a (DEFAULT_TABLE table=default.Table_d18d308917fc44cbb7f0af85912d01c8 [id=5da9ebbf87204a64ac540fd4efd33337]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.612056 1776 rpcz_store.cc:276] 1024 19:48:06.560099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.560292 (+ 193us) service_pool.cc:225] Handling call
1024 19:48:07.611833 (+1051541us) ts_tablet_manager.cc:1151] Deleting tablet 7c357a2a81064b3891afda6d252898b9
1024 19:48:07.611843 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.611914 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1051325}
I20251024 19:48:07.612391 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 510) took 1052 ms. Trace:
I20251024 19:48:07.612533 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14965e5ea7ae43f2b80986c58a2e9f2a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:07.618036 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 83b91a5d764449d5b82ef6d1ee0360f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83b91a5d764449d5b82ef6d1ee0360f5 already in progress: creating tablet
W20251024 19:48:07.618460 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7c357a2a81064b3891afda6d252898b9 because the tablet was not found. No further retry: Not found: Tablet not found: 7c357a2a81064b3891afda6d252898b9
I20251024 19:48:07.618511 1768 tablet_bootstrap.cc:492] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:07.618716 1768 ts_tablet_manager.cc:1403] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.555s user 0.006s sys 0.013s
I20251024 19:48:07.619592 1768 raft_consensus.cc:359] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.619781 1768 raft_consensus.cc:385] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.619874 1768 raft_consensus.cc:740] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.621219 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 857eed60ba2a4b309e86d88a0f093762 (table default.Table_b8e34cb629054ed9be6deaf4d30992a2 [id=d225099585ad4533b088821c639b5463]) successfully deleted
I20251024 19:48:07.612488 1776 rpcz_store.cc:276] 1024 19:48:06.560219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.560288 (+ 69us) service_pool.cc:225] Handling call
1024 19:48:07.612240 (+1051952us) ts_tablet_manager.cc:1151] Deleting tablet c04082bd1b444b468c177bef239701c6
1024 19:48:07.612247 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.612307 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1051694}
I20251024 19:48:07.622119 1776 tablet_replica.cc:333] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:07.622367 1776 raft_consensus.cc:2243] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:07.622589 1776 raft_consensus.cc:2272] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.623044 1767 tablet_bootstrap.cc:492] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:07.623149 1776 ts_tablet_manager.cc:1916] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.623152 1775 ts_tablet_manager.cc:1929] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:07.623273 1775 log.cc:1199] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/7c357a2a81064b3891afda6d252898b9
I20251024 19:48:07.623411 1767 ts_tablet_manager.cc:1403] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20251024 19:48:07.623709 1775 ts_tablet_manager.cc:1950] T 7c357a2a81064b3891afda6d252898b9 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:07.624468 1767 raft_consensus.cc:359] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.624775 1767 raft_consensus.cc:385] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.624905 1767 raft_consensus.cc:740] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.625254 1767 consensus_queue.cc:260] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.626030 1767 ts_tablet_manager.cc:1434] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251024 19:48:07.626214 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 45cff9bcc4924e0081518c284f074940 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45cff9bcc4924e0081518c284f074940 already in progress: creating tablet
I20251024 19:48:07.626513 1767 tablet_bootstrap.cc:492] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:48:07.626546 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2602335327fc489fa154056c5ef5138f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2602335327fc489fa154056c5ef5138f already in progress: creating tablet
W20251024 19:48:07.627739 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet c04082bd1b444b468c177bef239701c6 because the tablet was not found. No further retry: Not found: Tablet not found: c04082bd1b444b468c177bef239701c6
I20251024 19:48:07.628192 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 57103d6696d64848adee2fe976a22c93 (DEFAULT_TABLE table=default.Table_407e3b1465004f64bcabe7e2da5d4935 [id=c361c7f277c74818b31261921c102d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.628793 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57103d6696d64848adee2fe976a22c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.630529 1767 tablet_bootstrap.cc:654] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.633399 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 436) took 2998 ms. Trace:
I20251024 19:48:07.635331 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 57103d6696d64848adee2fe976a22c93 (DEFAULT_TABLE table=default.Table_407e3b1465004f64bcabe7e2da5d4935 [id=c361c7f277c74818b31261921c102d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.636497 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57103d6696d64848adee2fe976a22c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.639398 1185 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]). Expected version 1 got 0
I20251024 19:48:07.641208 1774 raft_consensus.cc:2243] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:07.641703 1775 rpcz_store.cc:276] 1024 19:48:04.634379 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.634451 (+ 72us) service_pool.cc:225] Handling call
1024 19:48:07.475975 (+2841524us) ts_tablet_manager.cc:1151] Deleting tablet 7c357a2a81064b3891afda6d252898b9
1024 19:48:07.475982 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.596369 (+120387us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.622989 (+ 26620us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.633270 (+ 10281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2841276}
I20251024 19:48:07.642392 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_adcb658ebf79468fb55f7635cf442d29" } modify_external_catalogs: true
I20251024 19:48:07.642637 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_adcb658ebf79468fb55f7635cf442d29" } modify_external_catalogs: true
I20251024 19:48:07.644979 2321 raft_consensus.cc:493] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.645170 2321 raft_consensus.cc:515] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.645882 2321 leader_election.cc:290] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.646039 1774 raft_consensus.cc:2272] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.646157 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 7c357a2a81064b3891afda6d252898b9 (table default.Table_c570270b7d7b4f5091c34e0550dd46ef [id=ed879934f5754b92b8cff234f3f66333]) successfully deleted
I20251024 19:48:07.646741 1774 ts_tablet_manager.cc:1916] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.646716 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 441) took 2966 ms. Trace:
I20251024 19:48:07.646837 1775 rpcz_store.cc:276] 1024 19:48:04.680033 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.680091 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:07.646540 (+2966449us) ts_tablet_manager.cc:1151] Deleting tablet 163ac050c8574615aef5061f563cfc0a
1024 19:48:07.646552 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.646673 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2966216}
W20251024 19:48:07.647284 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 163ac050c8574615aef5061f563cfc0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 163ac050c8574615aef5061f563cfc0a already in progress: creating tablet
I20251024 19:48:07.644614 1768 consensus_queue.cc:260] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.645380 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet a76d61c3464b41519d7657796bd853ff with delete_type TABLET_DATA_DELETED (replaced by tablet b240965605904533bd2af99f61aaef26) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:07.648743 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76d61c3464b41519d7657796bd853ff" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:07.648980 1509 raft_consensus.cc:2468] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:07.649449 1586 leader_election.cc:304] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:07.649844 2321 raft_consensus.cc:2804] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:07.649950 2321 raft_consensus.cc:493] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:07.650053 2321 raft_consensus.cc:3060] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.650645 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 442) took 2947 ms. Trace:
I20251024 19:48:07.650770 1775 rpcz_store.cc:276] 1024 19:48:04.702693 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.702769 (+ 76us) service_pool.cc:225] Handling call
1024 19:48:07.650094 (+2947325us) ts_tablet_manager.cc:1151] Deleting tablet f83b9290f00145a780318a43249eec12
1024 19:48:07.650104 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.650611 (+ 507us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2947109}
W20251024 19:48:07.651262 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet f83b9290f00145a780318a43249eec12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f83b9290f00145a780318a43249eec12 already in progress: creating tablet
W20251024 19:48:07.653005 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07e75c7f35544bec82cb994dea5ee4ef consensus state has no leader
I20251024 19:48:07.652912 2321 raft_consensus.cc:515] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.654122 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 443) took 2924 ms. Trace:
I20251024 19:48:07.655519 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76d61c3464b41519d7657796bd853ff" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:07.655705 1372 raft_consensus.cc:3060] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.656572 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e8f91be03f4e46d4b0847a86c58ffc6b" } modify_external_catalogs: true
I20251024 19:48:07.656862 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e8f91be03f4e46d4b0847a86c58ffc6b" } modify_external_catalogs: true
I20251024 19:48:07.656957 2321 leader_election.cc:290] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.657526 2372 consensus_queue.cc:1048] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:07.654121 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76d61c3464b41519d7657796bd853ff" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:07.657851 1510 raft_consensus.cc:3060] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.657733 1775 rpcz_store.cc:276] 1024 19:48:04.729555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.729625 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:07.653966 (+2924341us) ts_tablet_manager.cc:1151] Deleting tablet 9b89a90bb4ca447fbaf34fe39001c261
1024 19:48:07.653976 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.654077 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2924122}
W20251024 19:48:07.658334 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9b89a90bb4ca447fbaf34fe39001c261 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b89a90bb4ca447fbaf34fe39001c261 already in progress: creating tablet
I20251024 19:48:07.657505 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76d61c3464b41519d7657796bd853ff" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:07.660645 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 448) took 2879 ms. Trace:
I20251024 19:48:07.666326 1621 tablet_service.cc:1505] Processing CreateTablet for tablet b7074697285141e3a62f8f372d86a3c9 (DEFAULT_TABLE table=default.Table_31e350af7b2a4e38981c3db02a346e52 [id=231e82511d8142608ed6dd5f47f610fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.648037 1768 ts_tablet_manager.cc:1434] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20251024 19:48:07.668494 1768 tablet_bootstrap.cc:492] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:07.671193 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7074697285141e3a62f8f372d86a3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.661506 1775 rpcz_store.cc:276] 1024 19:48:04.780689 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.781430 (+ 741us) service_pool.cc:225] Handling call
1024 19:48:07.659532 (+2878102us) ts_tablet_manager.cc:1151] Deleting tablet 3cf4596b333244cc9e9e35a128721e4a
1024 19:48:07.659542 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.660595 (+ 1053us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2877803}
I20251024 19:48:07.673874 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 452) took 2841 ms. Trace:
I20251024 19:48:07.674551 1775 rpcz_store.cc:276] 1024 19:48:04.832766 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.833112 (+ 346us) service_pool.cc:225] Handling call
1024 19:48:07.673709 (+2840597us) ts_tablet_manager.cc:1151] Deleting tablet ebcc09d8e029451d8fb6b6601ad16087
1024 19:48:07.673720 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.673835 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2840251}
I20251024 19:48:07.675184 1775 tablet_replica.cc:333] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:07.675537 1775 raft_consensus.cc:2243] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:07.675864 1775 raft_consensus.cc:2272] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:07.679414 1485 tablet_service.cc:1505] Processing CreateTablet for tablet b7074697285141e3a62f8f372d86a3c9 (DEFAULT_TABLE table=default.Table_31e350af7b2a4e38981c3db02a346e52 [id=231e82511d8142608ed6dd5f47f610fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.680022 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7074697285141e3a62f8f372d86a3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.680465 1624 tablet_service.cc:1505] Processing CreateTablet for tablet c7ba26769fe143ad9da73f9f4a7920cf (DEFAULT_TABLE table=default.Table_ebaca4c94c0a4718bce68ebbc4feb98a [id=83b29c055d84488cbb6069dee672bbc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.680640 1767 tablet_bootstrap.cc:492] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:07.680837 1767 ts_tablet_manager.cc:1403] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.006s
I20251024 19:48:07.680974 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7ba26769fe143ad9da73f9f4a7920cf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:07.683717 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3cf4596b333244cc9e9e35a128721e4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cf4596b333244cc9e9e35a128721e4a already in progress: creating tablet
I20251024 19:48:07.663225 1510 raft_consensus.cc:2468] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:07.686456 1190 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_fdf3a5f81257407d8b5d28c7f7cab1b7 [id=0fa321e2d4d847e69435f6de429d8779] alter complete (version 2)
W20251024 19:48:07.687393 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4229bd266e04d04b970bc9fc760155f consensus state has no leader
W20251024 19:48:07.691601 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet ebcc09d8e029451d8fb6b6601ad16087 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebcc09d8e029451d8fb6b6601ad16087 already in progress: creating tablet
I20251024 19:48:07.692396 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_bc114bace73742c19c6d263e61d2bcdc" } modify_external_catalogs: true
I20251024 19:48:07.693511 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_bc114bace73742c19c6d263e61d2bcdc" } modify_external_catalogs: true
I20251024 19:48:07.693116 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5834ffa88ad9439a9052f00f4859d799" } new_table_name: "default.Table_34f366e7d1404f2e86acf32e02ae92ab" modify_external_catalogs: true
I20251024 19:48:07.695169 1766 tablet_bootstrap.cc:492] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:07.696044 1775 ts_tablet_manager.cc:1916] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:07.696331 2440 raft_consensus.cc:493] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.696290 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 14965e5ea7ae43f2b80986c58a2e9f2a (DEFAULT_TABLE table=default.Table_d18d308917fc44cbb7f0af85912d01c8 [id=5da9ebbf87204a64ac540fd4efd33337]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.696550 2440 raft_consensus.cc:515] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.697194 2440 leader_election.cc:290] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.701100 1484 tablet_service.cc:1505] Processing CreateTablet for tablet c7ba26769fe143ad9da73f9f4a7920cf (DEFAULT_TABLE table=default.Table_ebaca4c94c0a4718bce68ebbc4feb98a [id=83b29c055d84488cbb6069dee672bbc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.701894 1766 ts_tablet_manager.cc:1403] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.107s user 0.008s sys 0.003s
I20251024 19:48:07.702818 1766 raft_consensus.cc:359] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.703424 1766 raft_consensus.cc:385] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.703594 1766 raft_consensus.cc:740] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.703990 1766 consensus_queue.cc:260] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.706355 1372 raft_consensus.cc:2468] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:07.707772 1184 catalog_manager.cc:5649] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:07.708237 1184 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet ce0ef4443e574846a9f48fef3b893352 (table default.Table_f203b363de4c47fca0a0599140d18279 [id=ec57efb4d37e44f1b68045f58bc11c45]). Expected version 1 got 0
I20251024 19:48:07.709203 1766 ts_tablet_manager.cc:1434] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251024 19:48:07.709638 1766 tablet_bootstrap.cc:492] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:07.710201 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14965e5ea7ae43f2b80986c58a2e9f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.712422 1333 tablet_service.cc:1505] Processing CreateTablet for tablet c7ba26769fe143ad9da73f9f4a7920cf (DEFAULT_TABLE table=default.Table_ebaca4c94c0a4718bce68ebbc4feb98a [id=83b29c055d84488cbb6069dee672bbc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.713421 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 23a1b73314084848a4f28a042612f394 (DEFAULT_TABLE table=default.Table_fb1b7f0a2371441380ca5492cf1c6a3b [id=07e84ea89f3b46aa88fbe4086c1db019]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.713986 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7ba26769fe143ad9da73f9f4a7920cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.714012 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7503b22d9e12481b9af96385b9d76e50" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:07.714726 1350 tablet_service.cc:1505] Processing CreateTablet for tablet b7074697285141e3a62f8f372d86a3c9 (DEFAULT_TABLE table=default.Table_31e350af7b2a4e38981c3db02a346e52 [id=231e82511d8142608ed6dd5f47f610fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.715158 1766 tablet_bootstrap.cc:654] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.715478 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 14965e5ea7ae43f2b80986c58a2e9f2a (DEFAULT_TABLE table=default.Table_d18d308917fc44cbb7f0af85912d01c8 [id=5da9ebbf87204a64ac540fd4efd33337]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.716221 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 23a1b73314084848a4f28a042612f394 (DEFAULT_TABLE table=default.Table_fb1b7f0a2371441380ca5492cf1c6a3b [id=07e84ea89f3b46aa88fbe4086c1db019]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:07.716504 1447 leader_election.cc:343] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.716643 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a1b73314084848a4f28a042612f394. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.721932 1351 tablet_service.cc:1505] Processing CreateTablet for tablet f4229bd266e04d04b970bc9fc760155f (DEFAULT_TABLE table=default.Table_b20ffadae0f24d4e996d915faeef9ddb [id=f3635da6a9b04933a7024eb59254ad26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.724308 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 0ba845f3f8984585af370c3d39f0ebb9 (DEFAULT_TABLE table=default.Table_4f873edb1c0d444a88c3485b0aaa7623 [id=51f95e2cbe844938b4ce54900922d273]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.726593 1340 tablet_service.cc:1505] Processing CreateTablet for tablet e6a60513bb4d48f68dcad6b976fffa87 (DEFAULT_TABLE table=default.Table_ee8861d72c804704af20dbe501b5332d [id=ecb4c238c060444e83d5b56a9006cc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.727797 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 8af55c71122f4712ba1050e42f01ed83 (DEFAULT_TABLE table=default.Table_e8f91be03f4e46d4b0847a86c58ffc6b [id=99879a4ed2154f5c93cace009c97d401]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.729143 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 0e3a8d735a1741cabcabd64964f172fd (DEFAULT_TABLE table=default.Table_181f309333a24988954f97061ff12419 [id=18dad2fe2ca64f53bfe66b07b43aae76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.730373 1767 raft_consensus.cc:359] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.730705 1767 raft_consensus.cc:385] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:07.725785 1586 leader_election.cc:304] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:07.732913 1767 raft_consensus.cc:740] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:07.733246 1767 consensus_queue.cc:260] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:07.734972 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 47218.7us
I20251024 19:48:07.734985 2321 raft_consensus.cc:2804] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:07.734944 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7503b22d9e12481b9af96385b9d76e50" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:07.739581 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 23a1b73314084848a4f28a042612f394 (DEFAULT_TABLE table=default.Table_fb1b7f0a2371441380ca5492cf1c6a3b [id=07e84ea89f3b46aa88fbe4086c1db019]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.739933 1625 tablet_service.cc:1505] Processing CreateTablet for tablet f4229bd266e04d04b970bc9fc760155f (DEFAULT_TABLE table=default.Table_b20ffadae0f24d4e996d915faeef9ddb [id=f3635da6a9b04933a7024eb59254ad26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.738112 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet 55986287a8bb40c8bb28e536fb22fb27 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:07.740782 1628 tablet_service.cc:1505] Processing CreateTablet for tablet e6a60513bb4d48f68dcad6b976fffa87 (DEFAULT_TABLE table=default.Table_ee8861d72c804704af20dbe501b5332d [id=ecb4c238c060444e83d5b56a9006cc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.742394 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 0e3a8d735a1741cabcabd64964f172fd (DEFAULT_TABLE table=default.Table_181f309333a24988954f97061ff12419 [id=18dad2fe2ca64f53bfe66b07b43aae76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.743386 2321 raft_consensus.cc:697] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:07.743829 2321 consensus_queue.cc:237] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.746776 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 8af55c71122f4712ba1050e42f01ed83 (DEFAULT_TABLE table=default.Table_e8f91be03f4e46d4b0847a86c58ffc6b [id=99879a4ed2154f5c93cace009c97d401]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.759052 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 0ba845f3f8984585af370c3d39f0ebb9 (DEFAULT_TABLE table=default.Table_4f873edb1c0d444a88c3485b0aaa7623 [id=51f95e2cbe844938b4ce54900922d273]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.759750 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 93d84776848049c8a146998219986a84 (DEFAULT_TABLE table=default.Table_6cc41f0070cf4f22b3ce19c688e77c50 [id=374823b8c35b46d8a9dba817f54822b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.760027 1767 ts_tablet_manager.cc:1434] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.055s user 0.003s sys 0.001s
I20251024 19:48:07.761194 1339 tablet_service.cc:1505] Processing CreateTablet for tablet d610b007a0054067ab8a1278580a98d6 (DEFAULT_TABLE table=default.Table_7013504a3c6a4813991ac0e9022f53e1 [id=ac2c4408a8b14a5cb1252f5c8f811000]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.761206 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 2087e295ad4b4daa8eeebd49b43e08c8 (DEFAULT_TABLE table=default.Table_b82f8da2443f4f0f91dd2b762983abac [id=1990e068c1fe4affa1e4a6e76c39fdde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.763150 1348 tablet_service.cc:1505] Processing CreateTablet for tablet ca6b8783039a4f5fbbda65c129805642 (DEFAULT_TABLE table=default.Table_5c78cfb1ef614578accc5ec60079087b [id=767e92e851544b10a18bd5d5de23d9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.763744 2419 raft_consensus.cc:493] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.766252 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 2087e295ad4b4daa8eeebd49b43e08c8 (DEFAULT_TABLE table=default.Table_b82f8da2443f4f0f91dd2b762983abac [id=1990e068c1fe4affa1e4a6e76c39fdde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.766696 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a1b73314084848a4f28a042612f394. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.769111 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 452f77091adc4cf09de2231f2f6d845b (table default.Table_0fa2426ca885458797e98bedc44dec04 [id=9a49120c5b654676a63cd3966eaf31a9]). Expected version 3 got 0
W20251024 19:48:07.769914 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95f62010a24f432c84c89345f9b02b2f consensus state has no leader
I20251024 19:48:07.770377 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7d5114465a524d37b479ed5a6cebb938" } modify_external_catalogs: true
I20251024 19:48:07.770633 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7d5114465a524d37b479ed5a6cebb938" } modify_external_catalogs: true
W20251024 19:48:07.777693 1444 leader_election.cc:343] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.779057 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 5ace18d554f14acc892bc956c30e9979 (DEFAULT_TABLE table=default.Table_25c67815f6fc45c780e1a9fd8ae678f9 [id=a2f87693fc294ee5a7b0ec6d386c9b25]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.781337 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 93d84776848049c8a146998219986a84 (DEFAULT_TABLE table=default.Table_6cc41f0070cf4f22b3ce19c688e77c50 [id=374823b8c35b46d8a9dba817f54822b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.781879 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d84776848049c8a146998219986a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.783305 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 4ef1ab2b2d2c41f09c96c895942ef743 (DEFAULT_TABLE table=default.Table_c30ba0fd50f2482793bf9e2caecd448e [id=ac35e7dec4264486b886071805d9d388]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.785224 1629 tablet_service.cc:1505] Processing CreateTablet for tablet d610b007a0054067ab8a1278580a98d6 (DEFAULT_TABLE table=default.Table_7013504a3c6a4813991ac0e9022f53e1 [id=ac2c4408a8b14a5cb1252f5c8f811000]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.786218 1767 tablet_bootstrap.cc:492] T d692e1988d13460fb1d54e930e17fc9d P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:07.785486 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 5ace18d554f14acc892bc956c30e9979 (DEFAULT_TABLE table=default.Table_25c67815f6fc45c780e1a9fd8ae678f9 [id=a2f87693fc294ee5a7b0ec6d386c9b25]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.786657 1444 leader_election.cc:304] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:07.787029 2440 raft_consensus.cc:2749] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:07.787114 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 48000.7us
I20251024 19:48:07.787357 1491 tablet_service.cc:1505] Processing CreateTablet for tablet e6a60513bb4d48f68dcad6b976fffa87 (DEFAULT_TABLE table=default.Table_ee8861d72c804704af20dbe501b5332d [id=ecb4c238c060444e83d5b56a9006cc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.788959 1472 tablet_service.cc:1505] Processing CreateTablet for tablet 4ef1ab2b2d2c41f09c96c895942ef743 (DEFAULT_TABLE table=default.Table_c30ba0fd50f2482793bf9e2caecd448e [id=ac35e7dec4264486b886071805d9d388]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.789122 2020 raft_consensus.cc:493] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.789316 1473 tablet_service.cc:1505] Processing CreateTablet for tablet f4229bd266e04d04b970bc9fc760155f (DEFAULT_TABLE table=default.Table_b20ffadae0f24d4e996d915faeef9ddb [id=f3635da6a9b04933a7024eb59254ad26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.790123 2020 raft_consensus.cc:515] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.790287 1474 tablet_service.cc:1505] Processing CreateTablet for tablet d610b007a0054067ab8a1278580a98d6 (DEFAULT_TABLE table=default.Table_7013504a3c6a4813991ac0e9022f53e1 [id=ac2c4408a8b14a5cb1252f5c8f811000]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.791528 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 0ba845f3f8984585af370c3d39f0ebb9 (DEFAULT_TABLE table=default.Table_4f873edb1c0d444a88c3485b0aaa7623 [id=51f95e2cbe844938b4ce54900922d273]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.791922 1490 tablet_service.cc:1505] Processing CreateTablet for tablet ca6b8783039a4f5fbbda65c129805642 (DEFAULT_TABLE table=default.Table_5c78cfb1ef614578accc5ec60079087b [id=767e92e851544b10a18bd5d5de23d9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.792339 2440 raft_consensus.cc:493] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:07.792500 2440 raft_consensus.cc:515] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.793185 2440 leader_election.cc:290] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.793794 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 8af55c71122f4712ba1050e42f01ed83 (DEFAULT_TABLE table=default.Table_e8f91be03f4e46d4b0847a86c58ffc6b [id=99879a4ed2154f5c93cace009c97d401]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.795238 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 93d84776848049c8a146998219986a84 (DEFAULT_TABLE table=default.Table_6cc41f0070cf4f22b3ce19c688e77c50 [id=374823b8c35b46d8a9dba817f54822b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.764011 2419 raft_consensus.cc:515] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.797971 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba845f3f8984585af370c3d39f0ebb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.798260 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d84776848049c8a146998219986a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.798501 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2037fcca87b94b5fb31639b4d07307b0" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:07.790196 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 2087e295ad4b4daa8eeebd49b43e08c8 (DEFAULT_TABLE table=default.Table_b82f8da2443f4f0f91dd2b762983abac [id=1990e068c1fe4affa1e4a6e76c39fdde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.799660 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2037fcca87b94b5fb31639b4d07307b0" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:07.802206 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6a60513bb4d48f68dcad6b976fffa87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.802512 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca6b8783039a4f5fbbda65c129805642. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.803493 1187 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet b20a6845ae1645c0b51f096d471d4ccc (table default.Table_50e12cf5f39d4fa2983835ef0dbbc3d6 [id=201e5232763a4940a1288a56a695f71c]). Expected version 1 got 0
I20251024 19:48:07.803791 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4229bd266e04d04b970bc9fc760155f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.805734 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d610b007a0054067ab8a1278580a98d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.806169 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6a60513bb4d48f68dcad6b976fffa87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.809944 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet fca73ab61f0c47df9186459caf2b9a32 with delete_type TABLET_DATA_DELETED (replaced by tablet f83b9290f00145a780318a43249eec12) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:07.811043 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fca73ab61f0c47df9186459caf2b9a32" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:07.810163 1614 tablet_service.cc:1505] Processing CreateTablet for tablet ca6b8783039a4f5fbbda65c129805642 (DEFAULT_TABLE table=default.Table_5c78cfb1ef614578accc5ec60079087b [id=767e92e851544b10a18bd5d5de23d9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.812119 2419 leader_election.cc:290] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:07.816032 1586 leader_election.cc:343] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.816951 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fca73ab61f0c47df9186459caf2b9a32" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:07.818069 1648 raft_consensus.cc:2468] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:07.821481 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d610b007a0054067ab8a1278580a98d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.827015 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca6b8783039a4f5fbbda65c129805642. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.832038 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 9287f340fb554904986f6c18b7356701 (DEFAULT_TABLE table=default.Table_581140b54cae4ac09f8e9c1b940e4241 [id=51d77585367949b598ff4444aeb36878]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.832139 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 5ace18d554f14acc892bc956c30e9979 (DEFAULT_TABLE table=default.Table_25c67815f6fc45c780e1a9fd8ae678f9 [id=a2f87693fc294ee5a7b0ec6d386c9b25]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.832600 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 9287f340fb554904986f6c18b7356701 (DEFAULT_TABLE table=default.Table_581140b54cae4ac09f8e9c1b940e4241 [id=51d77585367949b598ff4444aeb36878]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.832038 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 4ef1ab2b2d2c41f09c96c895942ef743 (DEFAULT_TABLE table=default.Table_c30ba0fd50f2482793bf9e2caecd448e [id=ac35e7dec4264486b886071805d9d388]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.835179 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4229bd266e04d04b970bc9fc760155f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.851214 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ef1ab2b2d2c41f09c96c895942ef743. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.851132 1626 tablet_service.cc:1505] Processing CreateTablet for tablet ae0d26f010bd46d3a05ac50f966d5769 (DEFAULT_TABLE table=default.Table_72be3d9212c64e6dbe03acba1f12df24 [id=76cc66df6ce14bc8bf48e8c94b12351a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.851166 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 1e45eefeeff949139f9407f9f90fba2a (DEFAULT_TABLE table=default.Table_2fb3f9af398745dcafa14f615ef6a116 [id=1d6a83de1ebe48c9a0dea6b9d87170c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.851616 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae0d26f010bd46d3a05ac50f966d5769. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.855441 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9287f340fb554904986f6c18b7356701. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.857741 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e45eefeeff949139f9407f9f90fba2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.859673 1615 tablet_service.cc:1505] Processing CreateTablet for tablet bfe5aa5bd9844560874613d445ba5cc1 (DEFAULT_TABLE table=default.Table_63e00ebf66c2426a96902b5443eaa366 [id=9068dc2ff1234391ab50a0ecde28c69c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.860502 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7ba26769fe143ad9da73f9f4a7920cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.861008 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ef1ab2b2d2c41f09c96c895942ef743. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.861253 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9287f340fb554904986f6c18b7356701. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.861521 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2087e295ad4b4daa8eeebd49b43e08c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.861789 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ace18d554f14acc892bc956c30e9979. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.863070 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca6b8783039a4f5fbbda65c129805642. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.865711 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d84776848049c8a146998219986a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.869824 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d610b007a0054067ab8a1278580a98d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.874383 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8af55c71122f4712ba1050e42f01ed83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.883126 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e3a8d735a1741cabcabd64964f172fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.883618 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfe5aa5bd9844560874613d445ba5cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.883900 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2087e295ad4b4daa8eeebd49b43e08c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.885998 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8af55c71122f4712ba1050e42f01ed83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.887457 2020 leader_election.cc:290] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.888051 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ace18d554f14acc892bc956c30e9979. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.890105 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 0e3a8d735a1741cabcabd64964f172fd (DEFAULT_TABLE table=default.Table_181f309333a24988954f97061ff12419 [id=18dad2fe2ca64f53bfe66b07b43aae76]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:07.890256 1447 leader_election.cc:343] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:07.890499 1444 leader_election.cc:343] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:07.890628 1447 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 90302.1us
I20251024 19:48:07.890766 1444 leader_election.cc:304] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:07.891150 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 90293.7us
I20251024 19:48:07.891474 1444 leader_election.cc:304] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:07.892436 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fca73ab61f0c47df9186459caf2b9a32" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:07.898064 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2087e295ad4b4daa8eeebd49b43e08c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.907245 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 9287f340fb554904986f6c18b7356701 (DEFAULT_TABLE table=default.Table_581140b54cae4ac09f8e9c1b940e4241 [id=51d77585367949b598ff4444aeb36878]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.908013 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e3a8d735a1741cabcabd64964f172fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.909063 2440 raft_consensus.cc:2804] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:07.909703 2020 raft_consensus.cc:2749] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:07.909785 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 1e45eefeeff949139f9407f9f90fba2a (DEFAULT_TABLE table=default.Table_2fb3f9af398745dcafa14f615ef6a116 [id=1d6a83de1ebe48c9a0dea6b9d87170c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.911017 1775 ts_tablet_manager.cc:1929] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:07.911024 1480 tablet_service.cc:1505] Processing CreateTablet for tablet ae0d26f010bd46d3a05ac50f966d5769 (DEFAULT_TABLE table=default.Table_72be3d9212c64e6dbe03acba1f12df24 [id=76cc66df6ce14bc8bf48e8c94b12351a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.911852 1775 log.cc:1199] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/c04082bd1b444b468c177bef239701c6
I20251024 19:48:07.912343 1509 raft_consensus.cc:2468] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:07.914124 1766 tablet_bootstrap.cc:492] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:07.914319 1766 ts_tablet_manager.cc:1403] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.205s user 0.028s sys 0.032s
I20251024 19:48:07.914904 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e3a8d735a1741cabcabd64964f172fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.915859 2440 raft_consensus.cc:493] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:07.916005 2440 raft_consensus.cc:3060] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.919310 2440 raft_consensus.cc:515] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:07.920907 2440 leader_election.cc:290] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:07.929167 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8af55c71122f4712ba1050e42f01ed83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.929430 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ef1ab2b2d2c41f09c96c895942ef743. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.930286 1489 tablet_service.cc:1505] Processing CreateTablet for tablet bfe5aa5bd9844560874613d445ba5cc1 (DEFAULT_TABLE table=default.Table_63e00ebf66c2426a96902b5443eaa366 [id=9068dc2ff1234391ab50a0ecde28c69c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.931255 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfe5aa5bd9844560874613d445ba5cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.932214 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba845f3f8984585af370c3d39f0ebb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.932741 1775 ts_tablet_manager.cc:1950] T c04082bd1b444b468c177bef239701c6 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:07.934190 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 453) took 3100 ms. Trace:
I20251024 19:48:07.934336 1775 rpcz_store.cc:276] 1024 19:48:04.832933 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.833495 (+ 562us) service_pool.cc:225] Handling call
1024 19:48:07.675107 (+2841612us) ts_tablet_manager.cc:1151] Deleting tablet c04082bd1b444b468c177bef239701c6
1024 19:48:07.675118 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.722024 (+ 46906us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.910774 (+188750us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:07.933763 (+ 22989us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2841413}
I20251024 19:48:07.935567 1775 tablet_replica.cc:333] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:07.935848 1775 raft_consensus.cc:2243] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:07.936149 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae0d26f010bd46d3a05ac50f966d5769. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.942337 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 1e45eefeeff949139f9407f9f90fba2a (DEFAULT_TABLE table=default.Table_2fb3f9af398745dcafa14f615ef6a116 [id=1d6a83de1ebe48c9a0dea6b9d87170c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:07.944185 1586 leader_election.cc:304] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:07.952808 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4229bd266e04d04b970bc9fc760155f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.950022 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fca73ab61f0c47df9186459caf2b9a32" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:07.953729 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e45eefeeff949139f9407f9f90fba2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.953704 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fca73ab61f0c47df9186459caf2b9a32" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:07.959164 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a7d96771ce8440ebb201c4095b80c7c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:07.959928 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7d96771ce8440ebb201c4095b80c7c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.961061 2321 raft_consensus.cc:2804] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:07.962373 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6a973600640545a6947cf458bf8ca233"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:07.962988 2321 raft_consensus.cc:493] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251024 19:48:07.963173 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a973600640545a6947cf458bf8ca233 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:07.963254 1627 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b96fc2309274048bed7a35e4bf4bae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:07.964759 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6a4dbd8e2b264bbca433946dfb1890b7" } modify_external_catalogs: true
I20251024 19:48:07.964924 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6a4dbd8e2b264bbca433946dfb1890b7" } modify_external_catalogs: true
I20251024 19:48:07.967253 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet c04082bd1b444b468c177bef239701c6 (table default.Table_24200cd6f19d460cae963afb9b7f8724 [id=a2789faf4abe4c09b0a7515d941831a6]) successfully deleted
I20251024 19:48:07.962860 1775 raft_consensus.cc:2272] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:07.969898 1305 consensus_peers.cc:597] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a1f1892bd3b4776adc7c2519fcdf411. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:07.972153 1447 leader_election.cc:343] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.977394 2321 raft_consensus.cc:3060] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:07.982079 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14965e5ea7ae43f2b80986c58a2e9f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.984539 1768 tablet_bootstrap.cc:654] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:07.985805 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fca73ab61f0c47df9186459caf2b9a32" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:07.987915 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6a60513bb4d48f68dcad6b976fffa87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.988456 1775 ts_tablet_manager.cc:1916] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:07.989025 1444 consensus_peers.cc:597] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc1b6f0a377945089c50012f74d38564. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:07.977947 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet bc61fded84b24e00b299e5d342076708
I20251024 19:48:07.991928 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba845f3f8984585af370c3d39f0ebb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.994040 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ace18d554f14acc892bc956c30e9979. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:07.995113 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet b20a6845ae1645c0b51f096d471d4ccc (table default.Table_50e12cf5f39d4fa2983835ef0dbbc3d6 [id=201e5232763a4940a1288a56a695f71c]). Expected version 1 got 0
I20251024 19:48:07.996728 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c30ba0fd50f2482793bf9e2caecd448e" } new_table_name: "default.Table_70a5bb93919a463f858c20dfe378fe58" modify_external_catalogs: true
I20251024 19:48:07.997320 2020 raft_consensus.cc:493] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251024 19:48:07.997417 1447 leader_election.cc:343] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:07.987911 1335 tablet_service.cc:1505] Processing CreateTablet for tablet ae0d26f010bd46d3a05ac50f966d5769 (DEFAULT_TABLE table=default.Table_72be3d9212c64e6dbe03acba1f12df24 [id=76cc66df6ce14bc8bf48e8c94b12351a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.001817 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7074697285141e3a62f8f372d86a3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.002049 1348 tablet_service.cc:1505] Processing CreateTablet for tablet bfe5aa5bd9844560874613d445ba5cc1 (DEFAULT_TABLE table=default.Table_63e00ebf66c2426a96902b5443eaa366 [id=9068dc2ff1234391ab50a0ecde28c69c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.005851 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 8af55c71122f4712ba1050e42f01ed83
I20251024 19:48:08.008663 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 3754540ae21e4c65b5af3d0aa3f9189f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:08.010238 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d0aa4354692e4f0c8b105c7ab7603590"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.010881 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0aa4354692e4f0c8b105c7ab7603590 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.010946 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9287f340fb554904986f6c18b7356701. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:08.013921 1586 consensus_peers.cc:597] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:08.023298 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 8af55c71122f4712ba1050e42f01ed83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:08.024720 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e45eefeeff949139f9407f9f90fba2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.021488 1766 raft_consensus.cc:359] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.029253 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet c4d261f5f55949159aa1b72797e20fd5
I20251024 19:48:08.034035 2020 raft_consensus.cc:515] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:08.034579 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet c4d261f5f55949159aa1b72797e20fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:08.034943 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5f488d12a77948e68c3b5c1152e003f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.035573 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f488d12a77948e68c3b5c1152e003f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.035732 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet bc61fded84b24e00b299e5d342076708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:08.038434 1766 raft_consensus.cc:385] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.043314 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 8af55c71122f4712ba1050e42f01ed83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:08.044915 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3b8e8dbca79f4d72a2de51f105b209e4 consensus state has no leader
I20251024 19:48:08.046162 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet bc61fded84b24e00b299e5d342076708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:08.047148 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet bc61fded84b24e00b299e5d342076708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:08.047585 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfe5aa5bd9844560874613d445ba5cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.048740 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet 8af55c71122f4712ba1050e42f01ed83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:08.049302 1627 tablet_service.cc:1552] Processing DeleteTablet for tablet c4d261f5f55949159aa1b72797e20fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:08.050278 1650 raft_consensus.cc:1275] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Refusing update from remote peer 10a9536207e343f188e0ef709081d04e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 19:48:08.053930 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae0d26f010bd46d3a05ac50f966d5769. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.057471 1372 raft_consensus.cc:3060] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.058537 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f000460d66bd46a2bcae726de4f6fb1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.059180 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f000460d66bd46a2bcae726de4f6fb1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.072978 1766 raft_consensus.cc:740] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:08.074817 2419 consensus_queue.cc:1048] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20251024 19:48:08.067281 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a1b73314084848a4f28a042612f394. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:08.077442 1444 consensus_peers.cc:597] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50d919a0b3d74e9a97d136a84d50e24b. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:08.081455 2321 raft_consensus.cc:515] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.082808 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fca73ab61f0c47df9186459caf2b9a32" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:08.086004 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet c4d261f5f55949159aa1b72797e20fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:08.088052 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3259823b622461fa823841925b3d203" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:08.088403 2020 leader_election.cc:290] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:08.088932 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8916f1e163e3485db01f9d64a7db190c" } new_table_name: "default.Table_00d39085c7a9429eba70f98bba9edd11" modify_external_catalogs: true
I20251024 19:48:08.084908 1766 consensus_queue.cc:260] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.089628 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3259823b622461fa823841925b3d203" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:08.090123 1648 raft_consensus.cc:2393] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10a9536207e343f188e0ef709081d04e in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:08.090060 2334 consensus_queue.cc:1048] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20251024 19:48:08.090651 2321 leader_election.cc:290] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:08.091066 1586 leader_election.cc:343] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:08.095181 1767 tablet_bootstrap.cc:654] T d692e1988d13460fb1d54e930e17fc9d P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:08.096208 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 55986287a8bb40c8bb28e536fb22fb27
I20251024 19:48:08.096736 1183 catalog_manager.cc:6719] ReplaceTablet: tablet 55986287a8bb40c8bb28e536fb22fb27 deleted and replaced by tablet 2bb5f84c1f104b338be71cb55ae77929
I20251024 19:48:08.097018 1766 ts_tablet_manager.cc:1434] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.177s user 0.004s sys 0.000s
W20251024 19:48:08.097268 1444 consensus_peers.cc:597] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3b4e85f741849b2a6a3e8574396ff88. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:08.099680 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3bd0db5c2861405a9dd9d66731cd9a57" } new_table_name: "default.Table_ac4ce453efea4a268f70380f9ed99812" modify_external_catalogs: true
I20251024 19:48:08.099851 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 55986287a8bb40c8bb28e536fb22fb27 with delete_type TABLET_DATA_DELETED (replaced by tablet 2bb5f84c1f104b338be71cb55ae77929) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:08.108232 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet c1cdd5ec32ca49c1bfafaf3a24050e89
I20251024 19:48:08.108888 2419 consensus_queue.cc:1048] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251024 19:48:08.109692 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:08.109655 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fca73ab61f0c47df9186459caf2b9a32" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:08.114995 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet 55986287a8bb40c8bb28e536fb22fb27 with delete_type TABLET_DATA_DELETED (replaced by tablet 2bb5f84c1f104b338be71cb55ae77929) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:08.115749 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d66f2a41a7a84e57a7abd1ac8cc172b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.116436 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d66f2a41a7a84e57a7abd1ac8cc172b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.117169 1766 tablet_bootstrap.cc:492] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:08.119814 1776 ts_tablet_manager.cc:1929] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:08.119994 1776 log.cc:1199] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/5fd7726c532145c6b44ea9a37544510d
I20251024 19:48:08.121110 1372 raft_consensus.cc:1275] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Refusing update from remote peer 10a9536207e343f188e0ef709081d04e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 19:48:08.123921 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:08.125237 1768 tablet_bootstrap.cc:492] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:08.121522 1776 ts_tablet_manager.cc:1950] T 5fd7726c532145c6b44ea9a37544510d P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:08.126525 2372 raft_consensus.cc:493] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:08.127873 1510 raft_consensus.cc:2393] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
W20251024 19:48:08.127826 1447 leader_election.cc:343] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:08.129092 2372 raft_consensus.cc:515] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.131628 1768 ts_tablet_manager.cc:1403] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.459s user 0.002s sys 0.010s
I20251024 19:48:08.134137 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_114b0c5bec4643109970ce8f80add076"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.134768 1444 leader_election.cc:343] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:08.134826 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_114b0c5bec4643109970ce8f80add076 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.145193 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d601fa3e84af4fdc8768772e2283a86f" } new_table_name: "default.Table_88d611a9d596483bb568a4fbdcf71f17" modify_external_catalogs: true
I20251024 19:48:08.151650 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 57f522a825264ea7b215304dbb7aa2d5
I20251024 19:48:08.151577 1586 leader_election.cc:304] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:08.151610 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 55986287a8bb40c8bb28e536fb22fb27 with delete_type TABLET_DATA_DELETED (replaced by tablet 2bb5f84c1f104b338be71cb55ae77929) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:08.166181 2419 raft_consensus.cc:2749] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251024 19:48:08.172256 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet 57f522a825264ea7b215304dbb7aa2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:08.172292 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1312dfdab750454a8f412ce6a66284ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.172983 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1312dfdab750454a8f412ce6a66284ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.179984 2334 consensus_queue.cc:1048] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.048s
W20251024 19:48:08.181267 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ef1ab2b2d2c41f09c96c895942ef743 consensus state has no leader
I20251024 19:48:08.181541 1444 leader_election.cc:304] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:08.182054 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 84492.4us
I20251024 19:48:08.183367 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "452f77091adc4cf09de2231f2f6d845b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:08.183653 1648 raft_consensus.cc:2468] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:08.185150 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5f895593b7d0404c909c32f99e24d5b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.185806 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f895593b7d0404c909c32f99e24d5b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.186811 2372 leader_election.cc:290] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:08.187340 2419 raft_consensus.cc:493] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:08.187512 2419 raft_consensus.cc:515] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.190677 1305 leader_election.cc:304] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:08.191670 1444 leader_election.cc:304] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:08.191963 2431 raft_consensus.cc:2749] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251024 19:48:08.192562 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:08.193763 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 57f522a825264ea7b215304dbb7aa2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:08.188954 1472 tablet_service.cc:1552] Processing DeleteTablet for tablet 2037fcca87b94b5fb31639b4d07307b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d129bdc876e425aabcf1c99997e319a) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:08.200737 2334 raft_consensus.cc:2749] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:08.202672 1766 tablet_bootstrap.cc:654] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:08.202883 1768 raft_consensus.cc:359] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.205344 1768 raft_consensus.cc:385] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.205473 1768 raft_consensus.cc:740] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:08.205801 1768 consensus_queue.cc:260] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.211836 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b96fc2309274048bed7a35e4bf4bae2" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:08.213392 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 508) took 1653 ms. Trace:
I20251024 19:48:08.214835 1510 raft_consensus.cc:2468] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:08.215749 1586 leader_election.cc:304] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:08.215817 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1db3c2ecc936440985b132c9db2c56e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:08.216065 2419 leader_election.cc:290] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:08.216195 2321 raft_consensus.cc:2804] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251024 19:48:08.216406 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1db3c2ecc936440985b132c9db2c56e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.216403 2321 raft_consensus.cc:493] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:08.216758 2321 raft_consensus.cc:3060] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.217825 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9db467b0e3b745678279c3f288a7c988" } modify_external_catalogs: true
I20251024 19:48:08.218046 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9db467b0e3b745678279c3f288a7c988" } modify_external_catalogs: true
I20251024 19:48:08.213564 1776 rpcz_store.cc:276] 1024 19:48:06.559928 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.560578 (+ 650us) service_pool.cc:225] Handling call
1024 19:48:07.622027 (+1061449us) ts_tablet_manager.cc:1151] Deleting tablet 5fd7726c532145c6b44ea9a37544510d
1024 19:48:07.622037 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:07.669720 (+ 47683us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.107720 (+438000us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.213341 (+105621us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061171}
I20251024 19:48:08.218497 1776 tablet_replica.cc:333] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:08.218544 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 5fd7726c532145c6b44ea9a37544510d (table default.Table_999546c6f492460bb6dc96ab55b7d3ad [id=7dbbe53e70a547888a510250e0583e1d]) successfully deleted
I20251024 19:48:08.218717 1776 raft_consensus.cc:2243] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:08.218889 1776 raft_consensus.cc:2272] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:08.219388 1774 ts_tablet_manager.cc:1929] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:08.219597 1774 log.cc:1199] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/50d919a0b3d74e9a97d136a84d50e24b
I20251024 19:48:08.219660 1776 ts_tablet_manager.cc:1916] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:08.220749 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "452f77091adc4cf09de2231f2f6d845b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:08.221369 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_efcd2af1913c4a73ba5148272aa880aa" } modify_external_catalogs: true
I20251024 19:48:08.221590 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_efcd2af1913c4a73ba5148272aa880aa" } modify_external_catalogs: true
I20251024 19:48:08.223222 2321 raft_consensus.cc:515] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.224967 2321 leader_election.cc:290] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:08.229894 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce374bf694604250b3bbef15856ad01d has no consensus state
I20251024 19:48:08.232539 1767 tablet_bootstrap.cc:492] T d692e1988d13460fb1d54e930e17fc9d P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:08.232793 1767 ts_tablet_manager.cc:1403] T d692e1988d13460fb1d54e930e17fc9d P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.467s user 0.011s sys 0.008s
I20251024 19:48:08.235086 1767 raft_consensus.cc:359] T d692e1988d13460fb1d54e930e17fc9d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.235313 1767 raft_consensus.cc:385] T d692e1988d13460fb1d54e930e17fc9d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.235569 2334 consensus_queue.cc:1048] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20251024 19:48:08.237135 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 57f522a825264ea7b215304dbb7aa2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:08.235397 1767 raft_consensus.cc:740] T d692e1988d13460fb1d54e930e17fc9d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
W20251024 19:48:08.240185 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50cea450eda9485daba40bf875a1b90c consensus state has no leader
I20251024 19:48:08.240368 1767 consensus_queue.cc:260] T d692e1988d13460fb1d54e930e17fc9d P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.243498 2431 consensus_queue.cc:1048] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
W20251024 19:48:08.243969 1447 consensus_peers.cc:597] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:08.243991 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b96fc2309274048bed7a35e4bf4bae2" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:08.246495 2334 raft_consensus.cc:493] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:08.246876 1767 ts_tablet_manager.cc:1434] T d692e1988d13460fb1d54e930e17fc9d P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251024 19:48:08.247267 1767 tablet_bootstrap.cc:492] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:08.247874 1511 raft_consensus.cc:3060] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.249643 2334 raft_consensus.cc:515] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.250454 2334 leader_election.cc:290] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:08.250890 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_24a6bf47782a48f98a76f6bcda40352e" } new_table_name: "default.Table_61f1fbf85c954403a8374c6359cf52b9" modify_external_catalogs: true
I20251024 19:48:08.252200 1775 ts_tablet_manager.cc:1929] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:08.252394 1775 log.cc:1199] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/6ec5c7bf36874423b4790c6169da8bd0
I20251024 19:48:08.252583 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5c266c8c172d4d08847b98402d5d0ffb" } modify_external_catalogs: true
I20251024 19:48:08.252759 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5c266c8c172d4d08847b98402d5d0ffb" } modify_external_catalogs: true
I20251024 19:48:08.253118 1775 ts_tablet_manager.cc:1950] T 6ec5c7bf36874423b4790c6169da8bd0 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:08.253389 1774 ts_tablet_manager.cc:1950] T 50d919a0b3d74e9a97d136a84d50e24b P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:08.254843 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73ffe342f541447d8519ee24bc98a0f3" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:08.255326 1444 consensus_peers.cc:597] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 030e690ce4df4818a6450ae293b0ffbd. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:08.257506 1767 tablet_bootstrap.cc:654] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:08.262109 1776 ts_tablet_manager.cc:1929] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:08.262410 1776 log.cc:1199] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/d76410afd7ba4c73836f4791ef739056
I20251024 19:48:08.263046 1776 ts_tablet_manager.cc:1950] T d76410afd7ba4c73836f4791ef739056 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:08.264983 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 380) took 4471 ms. Trace:
I20251024 19:48:08.270200 1774 rpcz_store.cc:276] 1024 19:48:03.781986 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.782042 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:07.593374 (+3811332us) ts_tablet_manager.cc:1151] Deleting tablet 50d919a0b3d74e9a97d136a84d50e24b
1024 19:48:07.593382 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.052097 (+458715us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.214398 (+162301us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.253898 (+ 39500us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3811138}
I20251024 19:48:08.270715 1774 tablet_replica.cc:333] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:08.271888 1511 raft_consensus.cc:2468] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:08.272399 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 50d919a0b3d74e9a97d136a84d50e24b (table default.Table_d0ae44b1936f452d9c75bd4b888bfb30 [id=6652d9398f8e4495bdfbca917b1d0bab]) successfully deleted
I20251024 19:48:08.272464 1586 leader_election.cc:304] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:08.273053 2419 raft_consensus.cc:2804] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:08.277320 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73ffe342f541447d8519ee24bc98a0f3" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:08.283731 1444 consensus_peers.cc:597] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:08.284142 1444 leader_election.cc:343] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:08.286502 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 523) took 1488 ms. Trace:
I20251024 19:48:08.286643 1776 rpcz_store.cc:276] 1024 19:48:06.797794 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.797851 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:08.218419 (+1420568us) ts_tablet_manager.cc:1151] Deleting tablet d76410afd7ba4c73836f4791ef739056
1024 19:48:08.218431 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.242313 (+ 23882us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.261955 (+ 19642us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.286427 (+ 24472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1420336}
I20251024 19:48:08.287133 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet d76410afd7ba4c73836f4791ef739056 (table default.Table_34bbddd692a44f46a4925c957f4e8848 [id=6f9b0ed7552c4556afd0f59316831a91]) successfully deleted
I20251024 19:48:08.287649 2419 raft_consensus.cc:697] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:08.288089 2494 raft_consensus.cc:493] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:08.288455 2494 raft_consensus.cc:515] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.288462 2419 consensus_queue.cc:237] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.291108 1767 tablet_bootstrap.cc:492] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:08.291271 1767 ts_tablet_manager.cc:1403] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.001s
I20251024 19:48:08.292213 1767 raft_consensus.cc:359] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:08.292415 1767 raft_consensus.cc:385] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.292517 1767 raft_consensus.cc:740] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:08.292897 1767 consensus_queue.cc:260] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:08.294486 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 524) took 1496 ms. Trace:
I20251024 19:48:08.294598 1776 rpcz_store.cc:276] 1024 19:48:06.798286 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.798327 (+ 41us) service_pool.cc:225] Handling call
1024 19:48:08.294307 (+1495980us) ts_tablet_manager.cc:1151] Deleting tablet 6ec5c7bf36874423b4790c6169da8bd0
1024 19:48:08.294320 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.294413 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1495773}
I20251024 19:48:08.300575 2334 consensus_queue.cc:1048] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
W20251024 19:48:08.301729 1447 leader_election.cc:343] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:08.303187 1776 tablet_replica.cc:333] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:08.303882 1776 raft_consensus.cc:2243] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:08.304139 1776 raft_consensus.cc:2272] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:08.304852 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 454) took 3471 ms. Trace:
I20251024 19:48:08.307201 1776 ts_tablet_manager.cc:1916] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:08.307612 1775 rpcz_store.cc:276] 1024 19:48:04.833041 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.833728 (+ 687us) service_pool.cc:225] Handling call
1024 19:48:07.934837 (+3101109us) ts_tablet_manager.cc:1151] Deleting tablet 6ec5c7bf36874423b4790c6169da8bd0
1024 19:48:07.934846 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.101714 (+166868us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.241675 (+139961us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.304800 (+ 63125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":657,"tablet-delete.queue_time_us":3100893}
I20251024 19:48:08.309023 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 455) took 3370 ms. Trace:
I20251024 19:48:08.311038 2419 raft_consensus.cc:493] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:08.311254 2419 raft_consensus.cc:515] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.318819 2504 tablet.cc:1722] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251024 19:48:08.320864 1768 ts_tablet_manager.cc:1434] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.177s user 0.009s sys 0.001s
I20251024 19:48:08.325209 1768 tablet_bootstrap.cc:492] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:08.315531 1767 ts_tablet_manager.cc:1434] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20251024 19:48:08.319438 2419 leader_election.cc:290] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:08.315214 2494 leader_election.cc:290] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:08.330827 1767 tablet_bootstrap.cc:492] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:08.334703 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 452f77091adc4cf09de2231f2f6d845b (table default.Table_0fa2426ca885458797e98bedc44dec04 [id=9a49120c5b654676a63cd3966eaf31a9]). Expected version 3 got 0
I20251024 19:48:08.335594 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0116d95669da49668f8aa797d948aae3" } modify_external_catalogs: true
I20251024 19:48:08.335784 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0116d95669da49668f8aa797d948aae3" } modify_external_catalogs: true
I20251024 19:48:08.338465 1774 raft_consensus.cc:2243] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:08.309778 1447 leader_election.cc:304] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:08.317673 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "037c8cc1e53447109fafc047c02f196f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:08.338658 1774 pending_rounds.cc:70] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:08.338755 1774 pending_rounds.cc:77] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429325533777920 op_type: NO_OP noop_request { }
I20251024 19:48:08.338757 1510 raft_consensus.cc:2468] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:08.338862 1774 raft_consensus.cc:2889] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:08.338979 1774 raft_consensus.cc:2272] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:08.339339 1447 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 46523.3us
I20251024 19:48:08.339699 2440 raft_consensus.cc:2749] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:08.341111 2431 raft_consensus.cc:493] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:08.341298 2431 raft_consensus.cc:515] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.320536 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b20a6845ae1645c0b51f096d471d4ccc" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:08.341732 1511 raft_consensus.cc:2468] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:08.341943 2431 leader_election.cc:290] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:08.342258 1447 consensus_peers.cc:597] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:08.342733 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d692e1988d13460fb1d54e930e17fc9d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:08.343232 1447 leader_election.cc:343] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:08.342846 1775 rpcz_store.cc:276] 1024 19:48:04.938180 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.938246 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:08.308860 (+3370614us) ts_tablet_manager.cc:1151] Deleting tablet 6424b2347ef642feb38a9596f2d2be1d
1024 19:48:08.308874 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.308988 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3370343}
W20251024 19:48:08.343907 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 6424b2347ef642feb38a9596f2d2be1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6424b2347ef642feb38a9596f2d2be1d already in progress: creating tablet
I20251024 19:48:08.344235 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 456) took 3361 ms. Trace:
I20251024 19:48:08.344796 1774 ts_tablet_manager.cc:1916] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:08.344337 1775 rpcz_store.cc:276] 1024 19:48:04.982440 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.982515 (+ 75us) service_pool.cc:225] Handling call
1024 19:48:08.344055 (+3361540us) ts_tablet_manager.cc:1151] Deleting tablet ab488e4971874a16b2e052c84c250777
1024 19:48:08.344065 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.344199 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3361211}
W20251024 19:48:08.346303 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet ab488e4971874a16b2e052c84c250777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab488e4971874a16b2e052c84c250777 already in progress: creating tablet
I20251024 19:48:08.352268 1767 tablet_bootstrap.cc:654] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:08.354566 1444 consensus_peers.cc:597] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fd7726c532145c6b44ea9a37544510d. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:08.356185 1189 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]). Expected version 1 got 0
I20251024 19:48:08.356458 1189 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet ad055dc3387b414bb55d09b9683ba17e (table default.Table_cde3670bf0d04dc0b2049193bcee10d4 [id=c3de6937e93a48dcb88cabee59a09d32]). Expected version 1 got 0
I20251024 19:48:08.358506 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_289bdf22b5834ac78a2b65cd33fac990"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:08.358872 2489 raft_consensus.cc:2804] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251024 19:48:08.359158 2489 raft_consensus.cc:493] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10a9536207e343f188e0ef709081d04e)
I20251024 19:48:08.359323 2489 raft_consensus.cc:3060] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Advancing to term 2
I20251024 19:48:08.359273 2504 tablet.cc:1722] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:08.360194 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d692e1988d13460fb1d54e930e17fc9d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:08.360023 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_289bdf22b5834ac78a2b65cd33fac990 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.361027 1648 raft_consensus.cc:2468] T d692e1988d13460fb1d54e930e17fc9d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:08.361568 1444 leader_election.cc:304] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:08.361871 2440 raft_consensus.cc:2804] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:08.362004 2440 raft_consensus.cc:493] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:08.362104 2440 raft_consensus.cc:3060] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.362628 2489 raft_consensus.cc:515] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.323653 1766 tablet_bootstrap.cc:492] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:08.364257 1766 ts_tablet_manager.cc:1403] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.257s user 0.007s sys 0.007s
I20251024 19:48:08.364835 2440 raft_consensus.cc:515] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.325315 2334 consensus_queue.cc:1048] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:08.365689 2440 leader_election.cc:290] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:08.365918 1586 leader_election.cc:304] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:08.366137 1766 raft_consensus.cc:359] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:08.366268 1586 leader_election.cc:304] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:08.366444 1766 raft_consensus.cc:385] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.366627 1766 raft_consensus.cc:740] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:08.366523 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet fca73ab61f0c47df9186459caf2b9a32 with delete_type TABLET_DATA_DELETED (replaced by tablet f83b9290f00145a780318a43249eec12) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:08.367007 1766 consensus_queue.cc:260] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:08.367837 1775 tablet_replica.cc:333] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:08.373155 1775 raft_consensus.cc:2243] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:08.373826 1775 raft_consensus.cc:2272] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:08.374624 1775 ts_tablet_manager.cc:1916] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:08.379709 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e0dc7d4fed5448cb616af29e405e766 consensus state has no leader
I20251024 19:48:08.386054 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_67b56d43bd6145b48583bbe6e34c2f8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.387208 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67b56d43bd6145b48583bbe6e34c2f8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.388432 2494 raft_consensus.cc:2804] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:08.388669 2419 raft_consensus.cc:2804] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:08.394768 2494 raft_consensus.cc:493] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:08.395251 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d692e1988d13460fb1d54e930e17fc9d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:08.400527 2489 leader_election.cc:290] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:08.400900 1370 raft_consensus.cc:1240] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Rejecting Update request from peer 10a9536207e343f188e0ef709081d04e for earlier term 1. Current term is 2. Ops: []
I20251024 19:48:08.400906 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d692e1988d13460fb1d54e930e17fc9d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:08.401731 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "452f77091adc4cf09de2231f2f6d845b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:08.402235 1647 raft_consensus.cc:3060] T d692e1988d13460fb1d54e930e17fc9d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.402565 1768 tablet_bootstrap.cc:654] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:08.405802 1647 raft_consensus.cc:2468] T d692e1988d13460fb1d54e930e17fc9d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:08.407251 1776 ts_tablet_manager.cc:1929] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:08.407793 1776 log.cc:1199] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/0a12e415590a4c71aab4ad7d30412c83
I20251024 19:48:08.408281 1776 ts_tablet_manager.cc:1950] T 0a12e415590a4c71aab4ad7d30412c83 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:08.409197 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 525) took 1605 ms. Trace:
I20251024 19:48:08.409327 1776 rpcz_store.cc:276] 1024 19:48:06.803576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.803635 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:08.303067 (+1499432us) ts_tablet_manager.cc:1151] Deleting tablet 0a12e415590a4c71aab4ad7d30412c83
1024 19:48:08.303086 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.388779 (+ 85693us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.407075 (+ 18296us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.409151 (+ 2076us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1499170}
I20251024 19:48:08.409783 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 0a12e415590a4c71aab4ad7d30412c83 (table default.Table_894d6739340148a995dd849a205c19c3 [id=18ec057e920e4c21ae8cbe298058b6f3]) successfully deleted
I20251024 19:48:08.409960 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 526) took 1604 ms. Trace:
I20251024 19:48:08.410080 1776 rpcz_store.cc:276] 1024 19:48:06.805834 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.805887 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:08.409809 (+1603922us) ts_tablet_manager.cc:1151] Deleting tablet f88390f2534e4799a6eaa56aa4245e97
1024 19:48:08.409822 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.409928 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1603704}
I20251024 19:48:08.410640 1776 tablet_replica.cc:333] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:48:08.410753 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f88390f2534e4799a6eaa56aa4245e97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f88390f2534e4799a6eaa56aa4245e97 already in progress: creating tablet
I20251024 19:48:08.373369 2509 tablet.cc:1722] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251024 19:48:08.425088 1767 tablet_bootstrap.cc:492] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:08.425472 1767 ts_tablet_manager.cc:1403] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.003s
I20251024 19:48:08.427050 1767 raft_consensus.cc:359] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.427578 1767 raft_consensus.cc:385] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.427750 1767 raft_consensus.cc:740] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:08.430356 1774 ts_tablet_manager.cc:1929] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:08.431447 1774 log.cc:1199] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/f346a90a9efe442f82750336e3893deb
I20251024 19:48:08.433827 2494 raft_consensus.cc:3060] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.371709 2500 tablet.cc:1722] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:08.392141 1766 ts_tablet_manager.cc:1434] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20251024 19:48:08.442001 1766 tablet_bootstrap.cc:492] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:08.441741 1767 consensus_queue.cc:260] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.441903 2419 raft_consensus.cc:493] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251024 19:48:08.396808 1447 leader_election.cc:343] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:08.445940 1444 leader_election.cc:304] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:08.446295 2020 raft_consensus.cc:2804] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:08.446470 2020 raft_consensus.cc:697] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:08.447151 2020 consensus_queue.cc:237] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.448568 2419 raft_consensus.cc:3060] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.450285 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "452f77091adc4cf09de2231f2f6d845b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:08.450734 1767 ts_tablet_manager.cc:1434] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20251024 19:48:08.451857 1774 ts_tablet_manager.cc:1950] T f346a90a9efe442f82750336e3893deb P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:08.452486 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 381) took 4670 ms. Trace:
I20251024 19:48:08.452719 1774 rpcz_store.cc:276] 1024 19:48:03.782116 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.782184 (+ 68us) service_pool.cc:225] Handling call
1024 19:48:08.270648 (+4488464us) ts_tablet_manager.cc:1151] Deleting tablet f346a90a9efe442f82750336e3893deb
1024 19:48:08.270660 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.393616 (+122956us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.429114 (+ 35498us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.452422 (+ 23308us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67001,"tablet-delete.queue_time_us":4488173}
I20251024 19:48:08.453548 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet f346a90a9efe442f82750336e3893deb (table default.Table_673f2f271b3e4f9d9c5f6802a3646f8f [id=b79808d531b740d7abc0e6fc2d9ba219]) successfully deleted
I20251024 19:48:08.453652 1767 tablet_bootstrap.cc:492] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:08.455221 1308 leader_election.cc:304] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:08.455708 2487 raft_consensus.cc:2749] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251024 19:48:08.457473 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b96fc2309274048bed7a35e4bf4bae2" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:08.458007 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b96fc2309274048bed7a35e4bf4bae2" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:08.458763 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "037c8cc1e53447109fafc047c02f196f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:08.459124 1369 raft_consensus.cc:2468] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:08.459479 1774 tablet_replica.cc:333] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:08.460530 2419 raft_consensus.cc:515] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.462002 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b20a6845ae1645c0b51f096d471d4ccc" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:08.462587 1767 tablet_bootstrap.cc:654] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:08.464084 1768 tablet_bootstrap.cc:492] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:08.465155 1768 ts_tablet_manager.cc:1403] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.141s user 0.006s sys 0.000s
I20251024 19:48:08.466413 1768 raft_consensus.cc:359] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.466691 1768 raft_consensus.cc:385] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.466826 1768 raft_consensus.cc:740] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:08.467058 2501 raft_consensus.cc:493] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:08.467279 2501 raft_consensus.cc:515] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.467260 1768 consensus_queue.cc:260] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.468310 2501 leader_election.cc:290] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:08.468655 2419 leader_election.cc:290] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:08.469513 1768 ts_tablet_manager.cc:1434] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20251024 19:48:08.470098 1586 leader_election.cc:343] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:08.463289 1774 raft_consensus.cc:2243] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:08.471233 1767 tablet_bootstrap.cc:492] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:08.471385 1767 ts_tablet_manager.cc:1403] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
W20251024 19:48:08.471410 1586 consensus_peers.cc:597] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:08.472201 1767 raft_consensus.cc:359] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.472366 1767 raft_consensus.cc:385] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.472453 1767 raft_consensus.cc:740] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:08.472669 1767 consensus_queue.cc:260] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.473197 1767 ts_tablet_manager.cc:1434] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251024 19:48:08.473353 1586 leader_election.cc:343] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:08.474083 1767 tablet_bootstrap.cc:492] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:08.474406 1768 tablet_bootstrap.cc:492] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:08.476207 1586 leader_election.cc:343] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:08.476616 1774 raft_consensus.cc:2272] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:08.478030 2494 raft_consensus.cc:515] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.479234 2494 leader_election.cc:290] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:08.481693 1776 raft_consensus.cc:2243] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:08.482080 1776 pending_rounds.cc:70] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:08.482164 1776 pending_rounds.cc:77] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429336484769792 op_type: NO_OP noop_request { }
I20251024 19:48:08.482271 1776 raft_consensus.cc:2889] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:08.482640 1776 raft_consensus.cc:2272] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:08.483911 1776 ts_tablet_manager.cc:1916] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:08.490535 1774 ts_tablet_manager.cc:1916] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:08.493865 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b20a6845ae1645c0b51f096d471d4ccc" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:08.496259 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3754540ae21e4c65b5af3d0aa3f9189f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:08.496490 1369 raft_consensus.cc:2468] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:08.500155 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1c5fc102a45e49948a8157aa0b5e6949" } new_table_name: "default.Table_e5c532eb7fc241039c7e06e4757ea011" modify_external_catalogs: true
I20251024 19:48:08.501466 1768 tablet_bootstrap.cc:654] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:08.503058 1447 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 51118.3us
W20251024 19:48:08.503329 1586 leader_election.cc:343] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:08.503521 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a1366192b3134e3abfc231729cb42344" } modify_external_catalogs: true
I20251024 19:48:08.503752 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a1366192b3134e3abfc231729cb42344" } modify_external_catalogs: true
I20251024 19:48:08.504313 1766 tablet_bootstrap.cc:654] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:08.505115 1586 leader_election.cc:304] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:08.505611 2494 raft_consensus.cc:2804] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:08.505795 2494 raft_consensus.cc:493] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:08.505977 2494 raft_consensus.cc:3060] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:08.511013 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c887b0df24c45d894ca10d2eda71444 consensus state has no leader
I20251024 19:48:08.502945 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "037c8cc1e53447109fafc047c02f196f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:08.511330 1369 raft_consensus.cc:3060] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.514727 1776 ts_tablet_manager.cc:1929] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:08.514873 1776 log.cc:1199] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/9fffc42ddda14bd28800917ff2f0ced5
I20251024 19:48:08.515362 1369 raft_consensus.cc:2468] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:08.515527 1776 ts_tablet_manager.cc:1950] T 9fffc42ddda14bd28800917ff2f0ced5 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:08.515973 1586 leader_election.cc:304] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:08.516369 2419 raft_consensus.cc:2804] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:08.516559 2419 raft_consensus.cc:697] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:08.516803 1767 tablet_bootstrap.cc:654] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:08.517032 2419 consensus_queue.cc:237] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.526117 2494 raft_consensus.cc:515] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.526331 1184 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 037c8cc1e53447109fafc047c02f196f (table default.Table_e87be853efba49aeb50e71270beac600 [id=5bc052b3e0f04699b4b391bcec33617d]). Expected version 1 got 0
I20251024 19:48:08.526579 1184 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 452f77091adc4cf09de2231f2f6d845b (table default.Table_0fa2426ca885458797e98bedc44dec04 [id=9a49120c5b654676a63cd3966eaf31a9]). Expected version 3 got 0
I20251024 19:48:08.528200 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet a48249f1824a40b087b80a563816d485 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:08.532260 1188 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet d692e1988d13460fb1d54e930e17fc9d (table default.Table_b09229e0e1254adeacfed952befaa65a [id=67a0be12c85648bdb4515a096db21d3b]). Expected version 1 got 0
I20251024 19:48:08.535012 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 2d39eb2f825b417a9361c6f4876e0cd7 (DEFAULT_TABLE table=default.Table_ad7f12b3760d494db551de925da842df [id=40754e11103c497b8c6e4d45d0e38bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.535233 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 2d39eb2f825b417a9361c6f4876e0cd7 (DEFAULT_TABLE table=default.Table_ad7f12b3760d494db551de925da842df [id=40754e11103c497b8c6e4d45d0e38bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.535466 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d39eb2f825b417a9361c6f4876e0cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.535660 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d39eb2f825b417a9361c6f4876e0cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.538002 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 3754540ae21e4c65b5af3d0aa3f9189f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:08.540117 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 659313c09c0b4e30bf8e8439f05b90ef (DEFAULT_TABLE table=default.Table_b2f56bfc63f44d61b5edaa6bf59b7009 [id=d79b7229fb484ec6ab357c38bdc9ce82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.540462 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 659313c09c0b4e30bf8e8439f05b90ef (DEFAULT_TABLE table=default.Table_b2f56bfc63f44d61b5edaa6bf59b7009 [id=d79b7229fb484ec6ab357c38bdc9ce82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.540589 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 659313c09c0b4e30bf8e8439f05b90ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.540875 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 659313c09c0b4e30bf8e8439f05b90ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.543394 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 54bc7c7764584e11948d2831a4e5e0b6 (DEFAULT_TABLE table=default.Table_3b7821f2ba4a40a6a1d6c6ae777feb70 [id=4d97edd2b4904d2f9327a838e813ed50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.543849 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54bc7c7764584e11948d2831a4e5e0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.547885 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 54bc7c7764584e11948d2831a4e5e0b6 (DEFAULT_TABLE table=default.Table_3b7821f2ba4a40a6a1d6c6ae777feb70 [id=4d97edd2b4904d2f9327a838e813ed50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.548106 1336 tablet_service.cc:1505] Processing CreateTablet for tablet c0f231f739b64f9fb6a726589a213146 (DEFAULT_TABLE table=default.Table_6651e9c63f1d4f19b3d529bff417377b [id=3aa7246f88fc48249a2e76cb1bd83ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.548364 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54bc7c7764584e11948d2831a4e5e0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.548605 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f231f739b64f9fb6a726589a213146. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.558216 2334 consensus_queue.cc:1059] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 0, Time since last communication: 0.054s
I20251024 19:48:08.559944 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "037c8cc1e53447109fafc047c02f196f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:08.560146 1508 raft_consensus.cc:3060] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.562465 1334 tablet_service.cc:1505] Processing CreateTablet for tablet fc398cfc57084d6295a56dd3527fbfa5 (DEFAULT_TABLE table=default.Table_11dceac7c7394c128fa1ede51ee9fdab [id=56dd9b94cc7a4dedaadf271fda7e01e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.563078 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc398cfc57084d6295a56dd3527fbfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.563195 1508 raft_consensus.cc:2468] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:08.563934 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b20a6845ae1645c0b51f096d471d4ccc" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:08.563568 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3754540ae21e4c65b5af3d0aa3f9189f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:08.564997 2020 raft_consensus.cc:3055] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Stepping down as leader of term 1
I20251024 19:48:08.569013 1510 raft_consensus.cc:2468] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:08.569322 1511 raft_consensus.cc:3060] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.570845 1620 tablet_service.cc:1505] Processing CreateTablet for tablet c0f231f739b64f9fb6a726589a213146 (DEFAULT_TABLE table=default.Table_6651e9c63f1d4f19b3d529bff417377b [id=3aa7246f88fc48249a2e76cb1bd83ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.571322 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f231f739b64f9fb6a726589a213146. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.574963 1348 tablet_service.cc:1505] Processing CreateTablet for tablet ce374bf694604250b3bbef15856ad01d (DEFAULT_TABLE table=default.Table_00d39085c7a9429eba70f98bba9edd11 [id=4c4ba9a107c341faab372135e62adbed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.579269 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 1af368f347774270bca1a37c65d994ff (DEFAULT_TABLE table=default.Table_5b894129f4b642909eeb0fe89ec44dea [id=0a338ea8139a45bfb0eabf369e9c7b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.585394 1616 tablet_service.cc:1505] Processing CreateTablet for tablet ce374bf694604250b3bbef15856ad01d (DEFAULT_TABLE table=default.Table_00d39085c7a9429eba70f98bba9edd11 [id=4c4ba9a107c341faab372135e62adbed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.585721 1625 tablet_service.cc:1505] Processing CreateTablet for tablet fc398cfc57084d6295a56dd3527fbfa5 (DEFAULT_TABLE table=default.Table_11dceac7c7394c128fa1ede51ee9fdab [id=56dd9b94cc7a4dedaadf271fda7e01e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.586350 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3754540ae21e4c65b5af3d0aa3f9189f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:08.586604 1369 raft_consensus.cc:3060] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.588459 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1af368f347774270bca1a37c65d994ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:08.589905 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 7e0dc7d4fed5448cb616af29e405e766 (table default.Table_88d611a9d596483bb568a4fbdcf71f17 [id=f6226be8495d421bb26960fbed1fd190], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7e0dc7d4fed5448cb616af29e405e766 consensus state has no leader
I20251024 19:48:08.591107 1369 raft_consensus.cc:2468] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:08.591498 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce374bf694604250b3bbef15856ad01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.593317 1586 leader_election.cc:304] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:08.594854 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc398cfc57084d6295a56dd3527fbfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.596102 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce374bf694604250b3bbef15856ad01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.597054 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 1af368f347774270bca1a37c65d994ff (DEFAULT_TABLE table=default.Table_5b894129f4b642909eeb0fe89ec44dea [id=0a338ea8139a45bfb0eabf369e9c7b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.602200 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1af368f347774270bca1a37c65d994ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.604218 1774 ts_tablet_manager.cc:1929] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:08.615724 2419 raft_consensus.cc:2804] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
W20251024 19:48:08.610689 1305 consensus_peers.cc:597] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a12e415590a4c71aab4ad7d30412c83. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:08.629971 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 9229fa6bf13441fa894e7bb7f78d7038 (DEFAULT_TABLE table=default.Table_915650e3f7984a0d81ae2f98d2b1d0a6 [id=6d4415fcba5d41d5b77acd4c14e0ff78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.630220 1336 tablet_service.cc:1505] Processing CreateTablet for tablet bbbc3f1886e744ab9450a0f5776721d9 (DEFAULT_TABLE table=default.Table_0116d95669da49668f8aa797d948aae3 [id=297c701d61b04a2583d2815e8d838573]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.630662 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9229fa6bf13441fa894e7bb7f78d7038. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.631153 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbbc3f1886e744ab9450a0f5776721d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.632457 2419 raft_consensus.cc:697] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:08.633019 1350 tablet_service.cc:1505] Processing CreateTablet for tablet bd4deeaab67442898911dabfa7302492 (DEFAULT_TABLE table=default.Table_c3453ee396f645c79eb00b1777d54f12 [id=2ee6f2a05c354c3ab722357acace3a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.633155 1774 log.cc:1199] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/b5473699c1b647f8b86ee43fdb56bdfd
I20251024 19:48:08.634662 2419 consensus_queue.cc:237] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.637370 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 7e747880c4e1425ba749201e624a083e (DEFAULT_TABLE table=default.Table_d5fdf824dcd943138d9d3ab80f5bee04 [id=619e60f84da24d899ab2f69923f459b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.638424 1628 tablet_service.cc:1505] Processing CreateTablet for tablet bbbc3f1886e744ab9450a0f5776721d9 (DEFAULT_TABLE table=default.Table_0116d95669da49668f8aa797d948aae3 [id=297c701d61b04a2583d2815e8d838573]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.639014 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbbc3f1886e744ab9450a0f5776721d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.640295 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 0e37e6a790654d90a17bbb7d5d0f3ca5 (DEFAULT_TABLE table=default.Table_69f805dddbf3444cb3578c026af0da38 [id=0f194415b1594af8a9a41db0401ea97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.640843 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 9229fa6bf13441fa894e7bb7f78d7038 (DEFAULT_TABLE table=default.Table_915650e3f7984a0d81ae2f98d2b1d0a6 [id=6d4415fcba5d41d5b77acd4c14e0ff78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.633489 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd4deeaab67442898911dabfa7302492. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.639670 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 0e37e6a790654d90a17bbb7d5d0f3ca5 (DEFAULT_TABLE table=default.Table_69f805dddbf3444cb3578c026af0da38 [id=0f194415b1594af8a9a41db0401ea97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.644188 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e37e6a790654d90a17bbb7d5d0f3ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.644390 1774 ts_tablet_manager.cc:1950] T b5473699c1b647f8b86ee43fdb56bdfd P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:08.644833 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 382) took 4862 ms. Trace:
I20251024 19:48:08.645102 1774 rpcz_store.cc:276] 1024 19:48:03.782225 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.782304 (+ 79us) service_pool.cc:225] Handling call
1024 19:48:08.454026 (+4671722us) ts_tablet_manager.cc:1151] Deleting tablet b5473699c1b647f8b86ee43fdb56bdfd
1024 19:48:08.454040 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.532151 (+ 78111us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.604088 (+ 71937us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.644768 (+ 40680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4671501}
I20251024 19:48:08.645654 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet b5473699c1b647f8b86ee43fdb56bdfd (table default.Table_c700d3a5bb0c4963ace8624ae2ef882d [id=30691828be584171b7ee2472c9a25b4b]) successfully deleted
I20251024 19:48:08.646412 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 7e747880c4e1425ba749201e624a083e (DEFAULT_TABLE table=default.Table_d5fdf824dcd943138d9d3ab80f5bee04 [id=619e60f84da24d899ab2f69923f459b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.647106 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e747880c4e1425ba749201e624a083e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.650780 1617 tablet_service.cc:1505] Processing CreateTablet for tablet bd4deeaab67442898911dabfa7302492 (DEFAULT_TABLE table=default.Table_c3453ee396f645c79eb00b1777d54f12 [id=2ee6f2a05c354c3ab722357acace3a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.651559 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e747880c4e1425ba749201e624a083e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.651829 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e37e6a790654d90a17bbb7d5d0f3ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.658066 2020 raft_consensus.cc:740] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:08.659358 2020 consensus_queue.cc:260] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.659799 2020 raft_consensus.cc:3060] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Advancing to term 2
I20251024 19:48:08.661186 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd4deeaab67442898911dabfa7302492. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.662349 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9229fa6bf13441fa894e7bb7f78d7038. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.663184 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3754540ae21e4c65b5af3d0aa3f9189f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
W20251024 19:48:08.666857 1447 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 62119.7us
I20251024 19:48:08.670675 1510 raft_consensus.cc:3060] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.675112 2494 leader_election.cc:290] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:08.675583 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 64325d89db714554866e74922d103e34 (DEFAULT_TABLE table=default.Table_74e6a6d8b7a34049994d41a379a9a9f0 [id=7f87fee0587e4cb5a4f6f5afb3238334]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.679302 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 64325d89db714554866e74922d103e34 (DEFAULT_TABLE table=default.Table_74e6a6d8b7a34049994d41a379a9a9f0 [id=7f87fee0587e4cb5a4f6f5afb3238334]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.680136 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64325d89db714554866e74922d103e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.681723 1611 tablet_service.cc:1505] Processing CreateTablet for tablet ec2ffe32f2914ef59d5d77056fbb7d16 (DEFAULT_TABLE table=default.Table_18cdd8018c304deba24dd3429fdb48fc [id=3172f9d08e8649c289a02ad1cd61177d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.682168 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 7a65fc59de5346d6a9344c5310455e53 (DEFAULT_TABLE table=default.Table_9f30a26963bc4d0aa33b95bb45032870 [id=f2781f86dcb04f899e0eba25ae6f918b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.683018 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 069091f1c238487580baab38af6a70e2 (DEFAULT_TABLE table=default.Table_4798dd4a61224784b9963dd87de7a19a [id=998855bdba9a481ba7d844739c41fd83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.684320 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 069091f1c238487580baab38af6a70e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.687394 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 527) took 1878 ms. Trace:
I20251024 19:48:08.688815 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 069091f1c238487580baab38af6a70e2 (DEFAULT_TABLE table=default.Table_4798dd4a61224784b9963dd87de7a19a [id=998855bdba9a481ba7d844739c41fd83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.690770 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 80b673002d0742ef864bde7f1728683e (DEFAULT_TABLE table=default.Table_b519454f788a4f59bada834ba04bc981 [id=8235e7e3c14c4b0ea122d3c4124db253]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.695828 1333 tablet_service.cc:1505] Processing CreateTablet for tablet ec2ffe32f2914ef59d5d77056fbb7d16 (DEFAULT_TABLE table=default.Table_18cdd8018c304deba24dd3429fdb48fc [id=3172f9d08e8649c289a02ad1cd61177d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.705789 1776 rpcz_store.cc:276] 1024 19:48:06.809094 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.809141 (+ 47us) service_pool.cc:225] Handling call
1024 19:48:08.410576 (+1601435us) ts_tablet_manager.cc:1151] Deleting tablet 9fffc42ddda14bd28800917ff2f0ced5
1024 19:48:08.410585 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.481953 (+ 71368us) spinlock_profiling.cc:243] Waited 70.8 ms on lock 0x6160000e0e54. stack: 00007fea3c4169f6 00007fea3c4167b0 00007fea3b3f23ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fea482dc100 00007fea49629ae3 00007fea4c1e6287 00007fea4c2192d4 00007fea4c1f9d70 00007fea4c1f9afd 0000000000a8867c 00007fea3c45cd7f 00007fea3c46004b 00007fea3c45fec8
1024 19:48:08.492510 (+ 10557us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.514539 (+ 22029us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.687346 (+172807us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":162852224,"tablet-delete.queue_time_us":1601270}
I20251024 19:48:08.705507 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 2d39eb2f825b417a9361c6f4876e0cd7 (DEFAULT_TABLE table=default.Table_ad7f12b3760d494db551de925da842df [id=40754e11103c497b8c6e4d45d0e38bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.706259 1776 tablet_replica.cc:333] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:08.708159 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 9fffc42ddda14bd28800917ff2f0ced5 (table default.Table_64aaab6eda0a443fbb49223835af43e3 [id=52efb5f413d14db28314ecbb058a351f]) successfully deleted
I20251024 19:48:08.708882 1510 raft_consensus.cc:2468] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:08.709221 1511 raft_consensus.cc:2468] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:08.708884 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 9767f3fd8f3e48f6aa05bc054c1ddf67 (DEFAULT_TABLE table=default.Table_b9fe85eed58142c1a663db71a1f34756 [id=c5dfd7ab272241898b320da3ced9c10e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.712213 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 54bc7c7764584e11948d2831a4e5e0b6 (DEFAULT_TABLE table=default.Table_3b7821f2ba4a40a6a1d6c6ae777feb70 [id=4d97edd2b4904d2f9327a838e813ed50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.712224 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 7a65fc59de5346d6a9344c5310455e53 (DEFAULT_TABLE table=default.Table_9f30a26963bc4d0aa33b95bb45032870 [id=f2781f86dcb04f899e0eba25ae6f918b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.712574 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 659313c09c0b4e30bf8e8439f05b90ef (DEFAULT_TABLE table=default.Table_b2f56bfc63f44d61b5edaa6bf59b7009 [id=d79b7229fb484ec6ab357c38bdc9ce82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.712703 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 9767f3fd8f3e48f6aa05bc054c1ddf67 (DEFAULT_TABLE table=default.Table_b9fe85eed58142c1a663db71a1f34756 [id=c5dfd7ab272241898b320da3ced9c10e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.712893 1586 leader_election.cc:304] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:08.718106 2501 raft_consensus.cc:2804] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:08.720171 1484 tablet_service.cc:1505] Processing CreateTablet for tablet c0f231f739b64f9fb6a726589a213146 (DEFAULT_TABLE table=default.Table_6651e9c63f1d4f19b3d529bff417377b [id=3aa7246f88fc48249a2e76cb1bd83ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.720410 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d39eb2f825b417a9361c6f4876e0cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.721812 1350 tablet_service.cc:1505] Processing CreateTablet for tablet b95989b98c77472faa7bf940cfa60432 (DEFAULT_TABLE table=default.Table_27407bc1a5e3473abd1a59db49046843 [id=e8889ce5ceb3449da7568addd4a92bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.722193 1774 tablet_replica.cc:333] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:08.722277 1480 tablet_service.cc:1505] Processing CreateTablet for tablet 1af368f347774270bca1a37c65d994ff (DEFAULT_TABLE table=default.Table_5b894129f4b642909eeb0fe89ec44dea [id=0a338ea8139a45bfb0eabf369e9c7b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.722457 1774 raft_consensus.cc:2243] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:08.722572 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b673002d0742ef864bde7f1728683e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.722910 1774 raft_consensus.cc:2272] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:08.723157 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 659313c09c0b4e30bf8e8439f05b90ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.723248 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b95989b98c77472faa7bf940cfa60432. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.723716 1774 ts_tablet_manager.cc:1916] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:08.725761 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec2ffe32f2914ef59d5d77056fbb7d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.726111 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a65fc59de5346d6a9344c5310455e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.727728 1477 tablet_service.cc:1505] Processing CreateTablet for tablet fc398cfc57084d6295a56dd3527fbfa5 (DEFAULT_TABLE table=default.Table_11dceac7c7394c128fa1ede51ee9fdab [id=56dd9b94cc7a4dedaadf271fda7e01e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.728605 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9767f3fd8f3e48f6aa05bc054c1ddf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.729269 1488 tablet_service.cc:1505] Processing CreateTablet for tablet ce374bf694604250b3bbef15856ad01d (DEFAULT_TABLE table=default.Table_00d39085c7a9429eba70f98bba9edd11 [id=4c4ba9a107c341faab372135e62adbed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.729446 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54bc7c7764584e11948d2831a4e5e0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.729705 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f231f739b64f9fb6a726589a213146. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.730350 1620 tablet_service.cc:1505] Processing CreateTablet for tablet b95989b98c77472faa7bf940cfa60432 (DEFAULT_TABLE table=default.Table_27407bc1a5e3473abd1a59db49046843 [id=e8889ce5ceb3449da7568addd4a92bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.732403 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 7e747880c4e1425ba749201e624a083e (DEFAULT_TABLE table=default.Table_d5fdf824dcd943138d9d3ab80f5bee04 [id=619e60f84da24d899ab2f69923f459b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.736001 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1af368f347774270bca1a37c65d994ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.737814 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e747880c4e1425ba749201e624a083e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.740571 1775 ts_tablet_manager.cc:1929] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:08.741251 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64325d89db714554866e74922d103e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.741333 2501 raft_consensus.cc:697] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:08.741639 2501 consensus_queue.cc:237] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.742941 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec2ffe32f2914ef59d5d77056fbb7d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.743638 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3240c043bbbf492bb95cb5eda583a147"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:08.744225 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 069091f1c238487580baab38af6a70e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:08.744504 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3240c043bbbf492bb95cb5eda583a147 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.748389 1189 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet d692e1988d13460fb1d54e930e17fc9d (table default.Table_b09229e0e1254adeacfed952befaa65a [id=67a0be12c85648bdb4515a096db21d3b]). Expected version 1 got 0
I20251024 19:48:08.750021 1776 raft_consensus.cc:2243] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:08.750447 1776 raft_consensus.cc:2272] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:08.751813 1776 ts_tablet_manager.cc:1916] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:08.752527 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f186552a977c4a1e8002ece18abd9d07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.753149 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f186552a977c4a1e8002ece18abd9d07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.755551 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 80b673002d0742ef864bde7f1728683e (DEFAULT_TABLE table=default.Table_b519454f788a4f59bada834ba04bc981 [id=8235e7e3c14c4b0ea122d3c4124db253]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.755947 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b673002d0742ef864bde7f1728683e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.758425 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9767f3fd8f3e48f6aa05bc054c1ddf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.759183 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a65fc59de5346d6a9344c5310455e53. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:08.763259 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7503b22d9e12481b9af96385b9d76e50 consensus state has no leader
I20251024 19:48:08.763778 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d5fdf824dcd943138d9d3ab80f5bee04" } new_table_name: "default.Table_db62205c9d4b431bb48b4b75d0fcf8d1" modify_external_catalogs: true
I20251024 19:48:08.763906 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4c54e569156042fca5e9ab0feaedf632
I20251024 19:48:08.764704 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_69f805dddbf3444cb3578c026af0da38" } new_table_name: "default.Table_f19fe0b20aac4076a8bae6dc3fb1b502" modify_external_catalogs: true
I20251024 19:48:08.766268 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c54e569156042fca5e9ab0feaedf632 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:08.767630 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b95989b98c77472faa7bf940cfa60432. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.768313 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_379c43ee27bd4c74ae00adf4df216fa5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.768929 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_379c43ee27bd4c74ae00adf4df216fa5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.771687 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b240965605904533bd2af99f61aaef26
I20251024 19:48:08.772432 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet 38818c7997c84cedad5f9b09eef77105 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:08.772715 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet bbbc3f1886e744ab9450a0f5776721d9
I20251024 19:48:08.772825 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f51c56337ebb450aaf62dcc81ef63ae8
I20251024 19:48:08.773321 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet 23d4dd0b42d343cb9354ac698a8c761f from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:08.774407 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet b240965605904533bd2af99f61aaef26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:08.774677 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet bbbc3f1886e744ab9450a0f5776721d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:08.777948 1767 tablet_bootstrap.cc:492] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:08.783283 1767 ts_tablet_manager.cc:1403] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.305s user 0.001s sys 0.006s
I20251024 19:48:08.783807 1775 log.cc:1199] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/50d919a0b3d74e9a97d136a84d50e24b
I20251024 19:48:08.784317 1775 ts_tablet_manager.cc:1950] T 50d919a0b3d74e9a97d136a84d50e24b P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:08.784332 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 98dd8bacd8374f469799872859db4206
I20251024 19:48:08.785895 1768 tablet_bootstrap.cc:492] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:08.786563 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet b240965605904533bd2af99f61aaef26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:08.795162 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet f51c56337ebb450aaf62dcc81ef63ae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:08.795940 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce374bf694604250b3bbef15856ad01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.797011 1472 tablet_service.cc:1505] Processing CreateTablet for tablet bbbc3f1886e744ab9450a0f5776721d9 (DEFAULT_TABLE table=default.Table_0116d95669da49668f8aa797d948aae3 [id=297c701d61b04a2583d2815e8d838573]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.797760 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet f51c56337ebb450aaf62dcc81ef63ae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:08.798460 1767 raft_consensus.cc:359] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:08.798560 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 98dd8bacd8374f469799872859db4206 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:08.798774 1767 raft_consensus.cc:385] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.798903 1767 raft_consensus.cc:740] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:08.799306 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet bbbc3f1886e744ab9450a0f5776721d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:08.799226 1767 consensus_queue.cc:260] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:08.799955 1767 ts_tablet_manager.cc:1434] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.017s user 0.000s sys 0.003s
I20251024 19:48:08.800365 1767 tablet_bootstrap.cc:492] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:08.801539 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_181f309333a24988954f97061ff12419" } new_table_name: "default.Table_498bb6a3b91644e8831918efd611298d" modify_external_catalogs: true
I20251024 19:48:08.802899 1483 tablet_service.cc:1505] Processing CreateTablet for tablet bd4deeaab67442898911dabfa7302492 (DEFAULT_TABLE table=default.Table_c3453ee396f645c79eb00b1777d54f12 [id=2ee6f2a05c354c3ab722357acace3a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.803375 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6501382adb09425e8fa508d81fe78a24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:08.804121 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 98dd8bacd8374f469799872859db4206 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:08.804121 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6501382adb09425e8fa508d81fe78a24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.804927 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 9229fa6bf13441fa894e7bb7f78d7038 (DEFAULT_TABLE table=default.Table_915650e3f7984a0d81ae2f98d2b1d0a6 [id=6d4415fcba5d41d5b77acd4c14e0ff78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.806440 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_bd37f8f763d147b2b7656b63f72fd749"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.807032 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd37f8f763d147b2b7656b63f72fd749 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.811769 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 0e37e6a790654d90a17bbb7d5d0f3ca5 (DEFAULT_TABLE table=default.Table_69f805dddbf3444cb3578c026af0da38 [id=0f194415b1594af8a9a41db0401ea97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.812224 1485 tablet_service.cc:1505] Processing CreateTablet for tablet b95989b98c77472faa7bf940cfa60432 (DEFAULT_TABLE table=default.Table_27407bc1a5e3473abd1a59db49046843 [id=e8889ce5ceb3449da7568addd4a92bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.812659 1776 ts_tablet_manager.cc:1929] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:08.812903 1776 log.cc:1199] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/b6661b9e9a174849b6093d09cd8c1251
I20251024 19:48:08.814877 1776 ts_tablet_manager.cc:1950] T b6661b9e9a174849b6093d09cd8c1251 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:08.815536 1774 ts_tablet_manager.cc:1929] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:08.816314 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc398cfc57084d6295a56dd3527fbfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.816917 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e37e6a790654d90a17bbb7d5d0f3ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.820812 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b95989b98c77472faa7bf940cfa60432. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.821709 1767 tablet_bootstrap.cc:654] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:08.826722 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 528) took 2016 ms. Trace:
I20251024 19:48:08.826900 1776 rpcz_store.cc:276] 1024 19:48:06.809846 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.809890 (+ 44us) service_pool.cc:225] Handling call
1024 19:48:08.706183 (+1896293us) ts_tablet_manager.cc:1151] Deleting tablet b6661b9e9a174849b6093d09cd8c1251
1024 19:48:08.706194 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.755617 (+ 49423us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.794794 (+ 39177us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.826661 (+ 31867us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1896092}
I20251024 19:48:08.827332 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet b6661b9e9a174849b6093d09cd8c1251 (table default.Table_7dbe5740202642f1be406c4897a72541 [id=df94703d2472430387b81c028130d628]) successfully deleted
I20251024 19:48:08.830302 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 7a65fc59de5346d6a9344c5310455e53 (DEFAULT_TABLE table=default.Table_9f30a26963bc4d0aa33b95bb45032870 [id=f2781f86dcb04f899e0eba25ae6f918b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.831619 1774 log.cc:1199] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/a48249f1824a40b087b80a563816d485
I20251024 19:48:08.832301 1774 ts_tablet_manager.cc:1950] T a48249f1824a40b087b80a563816d485 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:08.832947 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbbc3f1886e744ab9450a0f5776721d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.833437 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 537) took 1619 ms. Trace:
I20251024 19:48:08.833633 1776 rpcz_store.cc:276] 1024 19:48:07.213776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.213913 (+ 137us) service_pool.cc:225] Handling call
1024 19:48:08.833121 (+1619208us) ts_tablet_manager.cc:1151] Deleting tablet fc7ccf1495924782b9b6ac1173bc73a1
1024 19:48:08.833132 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.833285 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1618941}
I20251024 19:48:08.834131 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a65fc59de5346d6a9344c5310455e53. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:08.834095 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet fc7ccf1495924782b9b6ac1173bc73a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc7ccf1495924782b9b6ac1173bc73a1 already in progress: creating tablet
I20251024 19:48:08.834800 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 459) took 3352 ms. Trace:
I20251024 19:48:08.834937 1775 rpcz_store.cc:276] 1024 19:48:05.458806 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.460665 (+ 1859us) service_pool.cc:225] Handling call
1024 19:48:08.346340 (+2885675us) ts_tablet_manager.cc:1151] Deleting tablet 50d919a0b3d74e9a97d136a84d50e24b
1024 19:48:08.367708 (+ 21368us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.418631 (+ 50923us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.732017 (+313386us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.811144 (+ 79127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2885396}
I20251024 19:48:08.835217 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 383) took 5013 ms. Trace:
I20251024 19:48:08.835325 1774 rpcz_store.cc:276] 1024 19:48:03.821728 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.821887 (+ 159us) service_pool.cc:225] Handling call
1024 19:48:08.697161 (+4875274us) ts_tablet_manager.cc:1151] Deleting tablet a48249f1824a40b087b80a563816d485
1024 19:48:08.697171 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.783706 (+ 86535us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.815337 (+ 31631us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.835176 (+ 19839us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4874906}
I20251024 19:48:08.835552 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 464) took 3302 ms. Trace:
I20251024 19:48:08.835698 1775 rpcz_store.cc:276] 1024 19:48:05.533079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.533136 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:08.835263 (+3302127us) ts_tablet_manager.cc:1151] Deleting tablet 15de0d8aa9e545c09233186bbe361b97
1024 19:48:08.835273 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.835438 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3301896}
I20251024 19:48:08.836024 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet a48249f1824a40b087b80a563816d485 (table default.Table_f96cb6877d844a50aacb91514a539fee [id=d9ab6489833b4989a69c975d734bcd2f]) successfully deleted
I20251024 19:48:08.825809 1768 ts_tablet_manager.cc:1403] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.313s user 0.007s sys 0.007s
I20251024 19:48:08.836149 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 465) took 3271 ms. Trace:
I20251024 19:48:08.836236 1775 rpcz_store.cc:276] 1024 19:48:05.565108 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.565408 (+ 300us) service_pool.cc:225] Handling call
1024 19:48:08.835944 (+3270536us) ts_tablet_manager.cc:1151] Deleting tablet 3e6f8764e9ae472b9ac35a032182aed1
1024 19:48:08.835952 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.836064 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3270183}
I20251024 19:48:08.836534 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 536) took 1622 ms. Trace:
I20251024 19:48:08.836606 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9229fa6bf13441fa894e7bb7f78d7038. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.836627 1776 rpcz_store.cc:276] 1024 19:48:07.213645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.244443 (+ 30798us) service_pool.cc:225] Handling call
1024 19:48:08.836373 (+1591930us) ts_tablet_manager.cc:1151] Deleting tablet 7deb97be82964ea0b2fe9caa867a6460
1024 19:48:08.836383 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.836477 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1591677}
I20251024 19:48:08.825992 1333 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c54e569156042fca5e9ab0feaedf632 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:08.837010 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7deb97be82964ea0b2fe9caa867a6460 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7deb97be82964ea0b2fe9caa867a6460 already in progress: creating tablet
I20251024 19:48:08.837242 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd4deeaab67442898911dabfa7302492. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.837667 1768 raft_consensus.cc:359] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.837766 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 539) took 1479 ms. Trace:
I20251024 19:48:08.837896 1768 raft_consensus.cc:385] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.837852 1776 rpcz_store.cc:276] 1024 19:48:07.357804 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.357858 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:08.837585 (+1479727us) ts_tablet_manager.cc:1151] Deleting tablet 0266cef4e2034303978d7456c47ea98c
1024 19:48:08.837594 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.837718 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1479475}
I20251024 19:48:08.837975 1768 raft_consensus.cc:740] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
W20251024 19:48:08.838261 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 0266cef4e2034303978d7456c47ea98c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0266cef4e2034303978d7456c47ea98c already in progress: creating tablet
I20251024 19:48:08.838661 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 471) took 3186 ms. Trace:
I20251024 19:48:08.838661 1768 consensus_queue.cc:260] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.838794 1775 rpcz_store.cc:276] 1024 19:48:05.651922 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.652635 (+ 713us) service_pool.cc:225] Handling call
1024 19:48:08.838510 (+3185875us) ts_tablet_manager.cc:1151] Deleting tablet 7c880a4af94d4041b8c7cf2066002ee3
1024 19:48:08.838518 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.838605 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3185653}
I20251024 19:48:08.839197 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 467) took 3187 ms. Trace:
I20251024 19:48:08.839262 1768 ts_tablet_manager.cc:1434] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:08.839358 1775 rpcz_store.cc:276] 1024 19:48:05.651386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.657038 (+ 5652us) service_pool.cc:225] Handling call
1024 19:48:08.839062 (+3182024us) ts_tablet_manager.cc:1151] Deleting tablet 505c30f0b2b749ce918b578903db3757
1024 19:48:08.839071 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.839149 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3181838}
I20251024 19:48:08.839776 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 466) took 3189 ms. Trace:
I20251024 19:48:08.839915 1775 rpcz_store.cc:276] 1024 19:48:05.650262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.650565 (+ 303us) service_pool.cc:225] Handling call
1024 19:48:08.839641 (+3189076us) ts_tablet_manager.cc:1151] Deleting tablet 3fd8aa1d9aca429dbfe46f472ebbf53b
1024 19:48:08.839650 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.839724 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3133846}
I20251024 19:48:08.840045 1768 tablet_bootstrap.cc:492] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:08.840363 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 472) took 3133 ms. Trace:
I20251024 19:48:08.842067 1768 tablet_bootstrap.cc:654] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:08.843943 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet a48249f1824a40b087b80a563816d485 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:08.844580 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f57b3f3656d347a3aee8146fd991ca1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.845172 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f57b3f3656d347a3aee8146fd991ca1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.849885 1774 tablet_replica.cc:333] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:08.850909 1774 raft_consensus.cc:2243] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:08.852309 2501 raft_consensus.cc:493] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:08.852486 2501 raft_consensus.cc:515] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.852680 1774 pending_rounds.cc:70] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:08.852785 1774 pending_rounds.cc:77] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429331947806720 op_type: NO_OP noop_request { }
I20251024 19:48:08.852905 1774 raft_consensus.cc:2889] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:08.853029 1774 raft_consensus.cc:2272] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:08.853333 2501 leader_election.cc:290] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:08.853678 1774 ts_tablet_manager.cc:1916] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:08.854029 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 540) took 1473 ms. Trace:
I20251024 19:48:08.854135 1776 rpcz_store.cc:276] 1024 19:48:07.380299 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.380359 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:08.853821 (+1473462us) ts_tablet_manager.cc:1151] Deleting tablet 4193750331874d7ba9bbe6279c3a8e37
1024 19:48:08.853837 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.853971 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1473199}
I20251024 19:48:08.854249 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad81d80697984ff1834a7d795e0f4b9b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:08.854619 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4193750331874d7ba9bbe6279c3a8e37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4193750331874d7ba9bbe6279c3a8e37 already in progress: creating tablet
W20251024 19:48:08.854894 1586 leader_election.cc:343] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:08.857182 1586 leader_election.cc:343] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: ad81d80697984ff1834a7d795e0f4b9b
I20251024 19:48:08.857322 1586 leader_election.cc:304] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:08.858350 2501 raft_consensus.cc:2749] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:08.859335 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 543) took 1432 ms. Trace:
I20251024 19:48:08.861676 1776 rpcz_store.cc:276] 1024 19:48:07.424342 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.424398 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:08.856976 (+1432578us) ts_tablet_manager.cc:1151] Deleting tablet d4b3a50db1714eb38cc0ea03f612fc95
1024 19:48:08.856985 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.857090 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1432109}
I20251024 19:48:08.863348 1767 tablet_bootstrap.cc:492] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:08.863831 1767 ts_tablet_manager.cc:1403] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.064s user 0.000s sys 0.009s
W20251024 19:48:08.863803 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d4b3a50db1714eb38cc0ea03f612fc95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4b3a50db1714eb38cc0ea03f612fc95 already in progress: creating tablet
I20251024 19:48:08.864859 1767 raft_consensus.cc:359] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.865108 1767 raft_consensus.cc:385] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.865235 1767 raft_consensus.cc:740] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:08.865545 1767 consensus_queue.cc:260] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.866946 1767 ts_tablet_manager.cc:1434] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:08.867312 1767 tablet_bootstrap.cc:492] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:08.868625 1768 tablet_bootstrap.cc:492] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:08.868846 1768 ts_tablet_manager.cc:1403] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20251024 19:48:08.868997 1767 tablet_bootstrap.cc:654] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:08.870224 1768 raft_consensus.cc:359] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.869395 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 542) took 1445 ms. Trace:
I20251024 19:48:08.870553 1768 raft_consensus.cc:385] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.870478 1776 rpcz_store.cc:276] 1024 19:48:07.423738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.423846 (+ 108us) service_pool.cc:225] Handling call
1024 19:48:08.869175 (+1445329us) ts_tablet_manager.cc:1151] Deleting tablet 7712392f200842c4b728e019d6a3cfb0
1024 19:48:08.869185 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.869300 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1444149}
I20251024 19:48:08.871026 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 544) took 1438 ms. Trace:
W20251024 19:48:08.871061 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7712392f200842c4b728e019d6a3cfb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7712392f200842c4b728e019d6a3cfb0 already in progress: creating tablet
I20251024 19:48:08.871119 1776 rpcz_store.cc:276] 1024 19:48:07.432115 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.432177 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:08.870809 (+1438632us) ts_tablet_manager.cc:1151] Deleting tablet ce8c179e98944df0b510a0e8daf3920f
1024 19:48:08.870818 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.870971 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438320}
I20251024 19:48:08.871522 1776 tablet_replica.cc:333] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:08.871709 1776 raft_consensus.cc:2243] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:08.871703 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet ce8c179e98944df0b510a0e8daf3920f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce8c179e98944df0b510a0e8daf3920f already in progress: creating tablet
I20251024 19:48:08.871891 1776 raft_consensus.cc:2272] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:08.872187 1768 raft_consensus.cc:740] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:08.872391 1776 ts_tablet_manager.cc:1916] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:08.872679 1768 consensus_queue.cc:260] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.873522 1768 ts_tablet_manager.cc:1434] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:08.874053 1768 tablet_bootstrap.cc:492] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:08.877091 1775 rpcz_store.cc:276] 1024 19:48:05.706449 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.707911 (+ 1462us) service_pool.cc:225] Handling call
1024 19:48:08.840215 (+3132304us) ts_tablet_manager.cc:1151] Deleting tablet 7da1601a87524ef1a8b5069ba6823843
1024 19:48:08.840224 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.840311 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3127033}
I20251024 19:48:08.882907 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 9767f3fd8f3e48f6aa05bc054c1ddf67 (DEFAULT_TABLE table=default.Table_b9fe85eed58142c1a663db71a1f34756 [id=c5dfd7ab272241898b320da3ced9c10e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.884483 1768 tablet_bootstrap.cc:654] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:08.884471 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9767f3fd8f3e48f6aa05bc054c1ddf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.886755 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 64325d89db714554866e74922d103e34 (DEFAULT_TABLE table=default.Table_74e6a6d8b7a34049994d41a379a9a9f0 [id=7f87fee0587e4cb5a4f6f5afb3238334]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.887215 1767 tablet_bootstrap.cc:492] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:08.887243 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64325d89db714554866e74922d103e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.887455 1767 ts_tablet_manager.cc:1403] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20251024 19:48:08.888656 1767 raft_consensus.cc:359] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.889037 1767 raft_consensus.cc:385] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.889231 1767 raft_consensus.cc:740] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:08.889758 1767 consensus_queue.cc:260] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.891870 1490 tablet_service.cc:1505] Processing CreateTablet for tablet ec2ffe32f2914ef59d5d77056fbb7d16 (DEFAULT_TABLE table=default.Table_18cdd8018c304deba24dd3429fdb48fc [id=3172f9d08e8649c289a02ad1cd61177d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.893606 1767 ts_tablet_manager.cc:1434] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251024 19:48:08.891588 1776 ts_tablet_manager.cc:1929] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:08.891922 2501 raft_consensus.cc:493] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:08.894658 2501 raft_consensus.cc:515] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.895506 2501 leader_election.cc:290] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:08.895948 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5368789df6024d7daa9ccc2257137d58" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:08.896093 1767 tablet_bootstrap.cc:492] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:08.896106 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5368789df6024d7daa9ccc2257137d58" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:08.896188 1509 raft_consensus.cc:2468] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:08.896312 1369 raft_consensus.cc:2468] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:08.895505 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 474) took 3126 ms. Trace:
I20251024 19:48:08.896907 1586 leader_election.cc:304] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:08.896886 1775 rpcz_store.cc:276] 1024 19:48:05.768811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.770749 (+ 1938us) service_pool.cc:225] Handling call
1024 19:48:08.877467 (+3106718us) ts_tablet_manager.cc:1151] Deleting tablet 505e4fb0755a4caaa462e5f5154bfee2
1024 19:48:08.877477 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.895444 (+ 17967us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3099850}
I20251024 19:48:08.897253 1775 tablet_replica.cc:333] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:08.897432 1775 raft_consensus.cc:2243] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:08.897778 2501 raft_consensus.cc:2804] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:08.897904 1775 raft_consensus.cc:2272] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:08.897981 2501 raft_consensus.cc:493] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:08.898080 2501 raft_consensus.cc:3060] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.898734 1775 ts_tablet_manager.cc:1916] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:08.898970 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec2ffe32f2914ef59d5d77056fbb7d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.900028 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 069091f1c238487580baab38af6a70e2 (DEFAULT_TABLE table=default.Table_4798dd4a61224784b9963dd87de7a19a [id=998855bdba9a481ba7d844739c41fd83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.900141 1776 log.cc:1199] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/4db17d9aff6542b6b94c09f872a4cceb
I20251024 19:48:08.902956 2501 raft_consensus.cc:515] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.903936 1776 ts_tablet_manager.cc:1950] T 4db17d9aff6542b6b94c09f872a4cceb P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:08.904742 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 545) took 1442 ms. Trace:
I20251024 19:48:08.905437 2501 leader_election.cc:290] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:08.905696 1776 rpcz_store.cc:276] 1024 19:48:07.462384 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.462456 (+ 72us) service_pool.cc:225] Handling call
1024 19:48:08.871433 (+1408977us) ts_tablet_manager.cc:1151] Deleting tablet 4db17d9aff6542b6b94c09f872a4cceb
1024 19:48:08.871442 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.881536 (+ 10094us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.891421 (+ 9885us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.904704 (+ 13283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1408709}
I20251024 19:48:08.906085 1776 tablet_replica.cc:333] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:08.906492 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 4db17d9aff6542b6b94c09f872a4cceb (table default.Table_ef3fdce598e74165bddac418dc92c20e [id=e4ee1d33b7e24827aa24950ea40465e4]) successfully deleted
I20251024 19:48:08.906563 1776 raft_consensus.cc:2243] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:08.906831 1776 pending_rounds.cc:70] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:08.906968 1776 pending_rounds.cc:77] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429338602803200 op_type: NO_OP noop_request { }
I20251024 19:48:08.907083 1776 raft_consensus.cc:2889] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:08.907631 1776 raft_consensus.cc:2272] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:08.907697 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5368789df6024d7daa9ccc2257137d58" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:08.907886 1369 raft_consensus.cc:3060] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:08.908232 1776 ts_tablet_manager.cc:1916] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:08.908600 1767 tablet_bootstrap.cc:654] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:08.911180 1369 raft_consensus.cc:2468] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:08.914139 1586 leader_election.cc:304] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:08.914903 2501 raft_consensus.cc:2804] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:08.915051 2501 raft_consensus.cc:697] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:08.915868 2501 consensus_queue.cc:237] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.918475 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 069091f1c238487580baab38af6a70e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.918529 1481 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c54e569156042fca5e9ab0feaedf632 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:08.920264 1444 consensus_peers.cc:597] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fffc42ddda14bd28800917ff2f0ced5. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:08.923605 1189 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_0fa2426ca885458797e98bedc44dec04 [id=9a49120c5b654676a63cd3966eaf31a9] alter complete (version 3)
I20251024 19:48:08.925226 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9ebd164c5e924a7eaff697af91c4c1a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.925868 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e37e6a790654d90a17bbb7d5d0f3ca5 consensus state has no leader
I20251024 19:48:08.926177 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_345aac1447b347f99accafba7777859a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.926702 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_345aac1447b347f99accafba7777859a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.927296 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fffc42ddda14bd28800917ff2f0ced5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:08.927395 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet 5368789df6024d7daa9ccc2257137d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:08.927606 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet b6661b9e9a174849b6093d09cd8c1251 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:08.927815 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ebd164c5e924a7eaff697af91c4c1a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.927822 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet ca0527af718f494a8bbcc66e7b325bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:08.927932 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e747880c4e1425ba749201e624a083e consensus state has no leader
I20251024 19:48:08.928332 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7837f9d39d7e4658a8945b4ddd2cf236"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:08.928777 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7837f9d39d7e4658a8945b4ddd2cf236 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.926194 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_be0ac003378749608afa5b5a6299c9f0" } new_table_name: "default.Table_361840e201d943c18014ec71a255cc9c" modify_external_catalogs: true
I20251024 19:48:08.944134 1776 ts_tablet_manager.cc:1929] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:08.944299 1776 log.cc:1199] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/a76d61c3464b41519d7657796bd853ff
I20251024 19:48:08.944911 1776 ts_tablet_manager.cc:1950] T a76d61c3464b41519d7657796bd853ff P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:08.946002 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 23d4dd0b42d343cb9354ac698a8c761f
W20251024 19:48:08.946031 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e3a8d735a1741cabcabd64964f172fd consensus state has no leader
I20251024 19:48:08.947193 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 23d4dd0b42d343cb9354ac698a8c761f with delete_type TABLET_DATA_DELETED (replaced by tablet fce7de9523fb4878a58aa8aef0c6b182) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:08.950136 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 547) took 1487 ms. Trace:
I20251024 19:48:08.950290 1776 rpcz_store.cc:276] 1024 19:48:07.463082 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.463141 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:08.905978 (+1442837us) ts_tablet_manager.cc:1151] Deleting tablet a76d61c3464b41519d7657796bd853ff
1024 19:48:08.905990 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.938113 (+ 32123us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.943942 (+ 5829us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.950093 (+ 6151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1442651}
I20251024 19:48:08.950820 1776 tablet_replica.cc:333] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:08.951069 1776 raft_consensus.cc:2243] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:08.951263 1776 raft_consensus.cc:2272] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:08.955978 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet 23d4dd0b42d343cb9354ac698a8c761f with delete_type TABLET_DATA_DELETED (replaced by tablet fce7de9523fb4878a58aa8aef0c6b182) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:08.956673 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b632acc1348e49f8a1eea019c007780c consensus state has no leader
I20251024 19:48:08.956900 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet f83b9290f00145a780318a43249eec12 from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:08.957221 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 78518.1us
I20251024 19:48:08.957273 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6eab640e6ecf469aa77cd34cbfa7b361"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:08.957660 1768 tablet_bootstrap.cc:492] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:08.957799 1768 ts_tablet_manager.cc:1403] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.000s
W20251024 19:48:08.957904 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6eab640e6ecf469aa77cd34cbfa7b361 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:08.957839 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 80b673002d0742ef864bde7f1728683e (DEFAULT_TABLE table=default.Table_b519454f788a4f59bada834ba04bc981 [id=8235e7e3c14c4b0ea122d3c4124db253]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:08.958719 1768 raft_consensus.cc:359] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.958946 1768 raft_consensus.cc:385] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.959004 1768 raft_consensus.cc:740] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:08.959537 2529 consensus_queue.cc:1048] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:08.960615 1776 ts_tablet_manager.cc:1916] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:08.961053 1768 consensus_queue.cc:260] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:08.962366 1186 catalog_manager.cc:6719] ReplaceTablet: tablet 23d4dd0b42d343cb9354ac698a8c761f deleted and replaced by tablet fce7de9523fb4878a58aa8aef0c6b182
I20251024 19:48:08.964218 1191 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet ad055dc3387b414bb55d09b9683ba17e (table default.Table_cde3670bf0d04dc0b2049193bcee10d4 [id=c3de6937e93a48dcb88cabee59a09d32]). Expected version 1 got 0
I20251024 19:48:08.964636 1481 tablet_service.cc:1552] Processing DeleteTablet for tablet b240965605904533bd2af99f61aaef26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:08.970347 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3240c043bbbf492bb95cb5eda583a147" } modify_external_catalogs: true
I20251024 19:48:08.970628 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3240c043bbbf492bb95cb5eda583a147" } modify_external_catalogs: true
I20251024 19:48:08.972604 1766 tablet_bootstrap.cc:492] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:08.972848 1766 ts_tablet_manager.cc:1403] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.531s user 0.008s sys 0.007s
I20251024 19:48:08.973228 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_67b56d43bd6145b48583bbe6e34c2f8c" } new_table_name: "default.Table_9e7fa6339ee24bab9d78fd7c318ed78d" modify_external_catalogs: true
I20251024 19:48:08.978150 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fffc42ddda14bd28800917ff2f0ced5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:08.978296 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_114b0c5bec4643109970ce8f80add076" } modify_external_catalogs: true
I20251024 19:48:08.978529 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_114b0c5bec4643109970ce8f80add076" } modify_external_catalogs: true
I20251024 19:48:08.978621 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c367e7fcf06a48fcbc2311c7a4998cac" } modify_external_catalogs: true
I20251024 19:48:08.979408 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c367e7fcf06a48fcbc2311c7a4998cac" } modify_external_catalogs: true
I20251024 19:48:08.980437 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_afee33bb95144492a2c62b19a101c0e3" } modify_external_catalogs: true
I20251024 19:48:08.980623 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_afee33bb95144492a2c62b19a101c0e3" } modify_external_catalogs: true
I20251024 19:48:08.981819 1186 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet fae4047f1b4d41d5a1586f4996d2fcf0
W20251024 19:48:08.982431 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f7b6d15f25d4c2a905da6597ba8d98a has no consensus state
I20251024 19:48:08.988507 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_86b9f6987402431196ad8064631c34bc" } modify_external_catalogs: true
I20251024 19:48:08.988790 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_86b9f6987402431196ad8064631c34bc" } modify_external_catalogs: true
I20251024 19:48:08.989326 1776 ts_tablet_manager.cc:1929] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:08.989508 1776 log.cc:1199] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/aebf76d4a66947ed9756643740418798
I20251024 19:48:08.990090 1776 ts_tablet_manager.cc:1950] T aebf76d4a66947ed9756643740418798 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:08.991614 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b673002d0742ef864bde7f1728683e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:08.991380 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5368789df6024d7daa9ccc2257137d58" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:08.995973 1766 raft_consensus.cc:359] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:08.996192 1766 raft_consensus.cc:385] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:08.996886 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 546) took 1533 ms. Trace:
I20251024 19:48:08.998821 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet bbbc3f1886e744ab9450a0f5776721d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:09.000835 1768 ts_tablet_manager.cc:1434] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20251024 19:48:09.000829 1509 raft_consensus.cc:3060] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:09.002883 1776 rpcz_store.cc:276] 1024 19:48:07.462970 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.463371 (+ 401us) service_pool.cc:225] Handling call
1024 19:48:08.950742 (+1487371us) ts_tablet_manager.cc:1151] Deleting tablet aebf76d4a66947ed9756643740418798
1024 19:48:08.950756 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.983481 (+ 32725us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.989184 (+ 5703us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:08.996821 (+ 7637us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1487180}
I20251024 19:48:09.003427 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet aebf76d4a66947ed9756643740418798 (table default.Table_d09f149249c44c50adcb71321c133f2d [id=665389c7bde54b7786edabcb91a9b33b]) successfully deleted
I20251024 19:48:09.004058 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 549) took 1505 ms. Trace:
I20251024 19:48:09.004190 1776 rpcz_store.cc:276] 1024 19:48:07.498097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.498154 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:09.003820 (+1505666us) ts_tablet_manager.cc:1151] Deleting tablet 4cac515c0a00403fb680454c1721be0a
1024 19:48:09.003835 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.003982 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1505320}
W20251024 19:48:09.004683 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4cac515c0a00403fb680454c1721be0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cac515c0a00403fb680454c1721be0a already in progress: creating tablet
I20251024 19:48:09.005627 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 554) took 1417 ms. Trace:
I20251024 19:48:09.005746 1776 rpcz_store.cc:276] 1024 19:48:07.587927 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.588096 (+ 169us) service_pool.cc:225] Handling call
1024 19:48:09.005388 (+1417292us) ts_tablet_manager.cc:1151] Deleting tablet 1295c99836d14a1cb0002e6c6aaee1eb
1024 19:48:09.005402 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.005522 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1417051}
W20251024 19:48:09.007368 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 1295c99836d14a1cb0002e6c6aaee1eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1295c99836d14a1cb0002e6c6aaee1eb already in progress: creating tablet
I20251024 19:48:09.006594 1766 raft_consensus.cc:740] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:09.010845 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet f51c56337ebb450aaf62dcc81ef63ae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:09.011310 1447 consensus_peers.cc:597] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:09.015264 1776 tablet_replica.cc:333] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:09.016665 1776 raft_consensus.cc:2243] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:09.010751 1766 consensus_queue.cc:260] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:09.019567 1776 raft_consensus.cc:2272] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:09.020705 1776 ts_tablet_manager.cc:1916] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:09.021101 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 50d919a0b3d74e9a97d136a84d50e24b (table default.Table_d0ae44b1936f452d9c75bd4b888bfb30 [id=6652d9398f8e4495bdfbca917b1d0bab]) successfully deleted
W20251024 19:48:09.021431 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 15de0d8aa9e545c09233186bbe361b97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15de0d8aa9e545c09233186bbe361b97 already in progress: creating tablet
I20251024 19:48:09.021528 1187 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 7503b22d9e12481b9af96385b9d76e50 (table default.Table_e5c532eb7fc241039c7e06e4757ea011 [id=9065c680f4da42b5a6e5ca335c8e2f3b]). Expected version 3 got 0
I20251024 19:48:09.021705 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 98dd8bacd8374f469799872859db4206 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:09.021742 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3e6f8764e9ae472b9ac35a032182aed1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e6f8764e9ae472b9ac35a032182aed1 already in progress: creating tablet
I20251024 19:48:09.021872 1187 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet a3259823b622461fa823841925b3d203 (table default.Table_ea0d3589c13a4ebeb950ddc737a9948b [id=350b37bf42194fcfbb660bffe83e47c4]). Expected version 4 got 0
W20251024 19:48:09.021982 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7c880a4af94d4041b8c7cf2066002ee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c880a4af94d4041b8c7cf2066002ee3 already in progress: creating tablet
I20251024 19:48:09.022174 1187 catalog_manager.cc:5649] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:09.022730 1187 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet b20a6845ae1645c0b51f096d471d4ccc (table default.Table_50e12cf5f39d4fa2983835ef0dbbc3d6 [id=201e5232763a4940a1288a56a695f71c]). Expected version 1 got 0
I20251024 19:48:09.023032 1187 catalog_manager.cc:5649] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:09.023502 1187 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 037c8cc1e53447109fafc047c02f196f (table default.Table_e87be853efba49aeb50e71270beac600 [id=5bc052b3e0f04699b4b391bcec33617d]). Expected version 1 got 0
W20251024 19:48:09.023597 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 505c30f0b2b749ce918b578903db3757 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 505c30f0b2b749ce918b578903db3757 already in progress: creating tablet
W20251024 19:48:09.023855 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fd8aa1d9aca429dbfe46f472ebbf53b already in progress: creating tablet
W20251024 19:48:09.024114 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7da1601a87524ef1a8b5069ba6823843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7da1601a87524ef1a8b5069ba6823843 already in progress: creating tablet
W20251024 19:48:09.024534 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 505e4fb0755a4caaa462e5f5154bfee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 505e4fb0755a4caaa462e5f5154bfee2 already in progress: creating tablet
W20251024 19:48:09.029659 1447 consensus_peers.cc:597] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:09.034637 1623 tablet_service.cc:1505] Processing CreateTablet for tablet c1f4bb2e3a5b406fb5b37403667be5d8 (DEFAULT_TABLE table=default.Table_289bdf22b5834ac78a2b65cd33fac990 [id=0f2a7b323e8d495e95b198eee2e7319d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.035233 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f4bb2e3a5b406fb5b37403667be5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.039600 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 0143a393711446fdbe6c798efa0c8936 (DEFAULT_TABLE table=default.Table_5f895593b7d0404c909c32f99e24d5b3 [id=186dcdefc76a4e62b86cd9b9e12a17f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.040300 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0143a393711446fdbe6c798efa0c8936. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.043318 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2bb5f84c1f104b338be71cb55ae77929
I20251024 19:48:09.044291 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet e68997931ede4b6398570a7d08edaf4d
I20251024 19:48:09.045006 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 95ae017a2e084ccabfa69c9a5befe0e7
I20251024 19:48:09.046918 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 2bb5f84c1f104b338be71cb55ae77929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:09.047333 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet e68997931ede4b6398570a7d08edaf4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:09.047715 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 95ae017a2e084ccabfa69c9a5befe0e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:09.050324 1186 catalog_manager.cc:5649] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:09.050904 1186 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet d692e1988d13460fb1d54e930e17fc9d (table default.Table_b09229e0e1254adeacfed952befaa65a [id=67a0be12c85648bdb4515a096db21d3b]). Expected version 1 got 0
I20251024 19:48:09.055110 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1606f259eb4f47678e56892bf267bf77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.055744 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1606f259eb4f47678e56892bf267bf77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.056831 1766 ts_tablet_manager.cc:1434] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.050s user 0.000s sys 0.003s
I20251024 19:48:09.060256 1766 tablet_bootstrap.cc:492] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:09.061064 1483 tablet_service.cc:1505] Processing CreateTablet for tablet c1f4bb2e3a5b406fb5b37403667be5d8 (DEFAULT_TABLE table=default.Table_289bdf22b5834ac78a2b65cd33fac990 [id=0f2a7b323e8d495e95b198eee2e7319d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.070871 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 1e6a588092a44688bc96f16ac00e39f9 (DEFAULT_TABLE table=default.Table_114b0c5bec4643109970ce8f80add076 [id=e3e6a0009da04ec49dc6d0d5fa0e39e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.071681 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 1e6a588092a44688bc96f16ac00e39f9
I20251024 19:48:09.073292 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e6a588092a44688bc96f16ac00e39f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:09.073820 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e6a588092a44688bc96f16ac00e39f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.075564 2522 raft_consensus.cc:493] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:09.078365 1623 tablet_service.cc:1505] Processing CreateTablet for tablet cd69fd2bba504eb398e483d6215a8eb9 (DEFAULT_TABLE table=default.Table_d66f2a41a7a84e57a7abd1ac8cc172b6 [id=43a9f0a67da54813a02f1a9aa9247ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.080113 1187 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_cde3670bf0d04dc0b2049193bcee10d4 [id=c3de6937e93a48dcb88cabee59a09d32] alter complete (version 1)
I20251024 19:48:09.081456 1509 raft_consensus.cc:2468] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:09.082899 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet aebf76d4a66947ed9756643740418798 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:09.083401 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 5368789df6024d7daa9ccc2257137d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:09.083804 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 73ffe342f541447d8519ee24bc98a0f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 505e4fb0755a4caaa462e5f5154bfee2) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:09.084236 1472 tablet_service.cc:1505] Processing CreateTablet for tablet 0143a393711446fdbe6c798efa0c8936 (DEFAULT_TABLE table=default.Table_5f895593b7d0404c909c32f99e24d5b3 [id=186dcdefc76a4e62b86cd9b9e12a17f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.085193 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet ad81d80697984ff1834a7d795e0f4b9b with delete_type TABLET_DATA_DELETED (replaced by tablet 7c880a4af94d4041b8c7cf2066002ee3) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:09.085775 1616 tablet_service.cc:1505] Processing CreateTablet for tablet aeeaf256b1404653b690c712d9ef7a6d (DEFAULT_TABLE table=default.Table_5f488d12a77948e68c3b5c1152e003f7 [id=5ffa582758154e6694adf569169c096b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:09.086305 2540 log.cc:927] Time spent T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20251024 19:48:09.087910 1620 tablet_service.cc:1505] Processing CreateTablet for tablet e5a9b28ff5674d5db8002b748e2733b5 (DEFAULT_TABLE table=default.Table_d0aa4354692e4f0c8b105c7ab7603590 [id=691cca934eed4899a85c7b8f147c74f4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:09.088037 1177 consensus_peers.cc:597] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c -> Peer ef8906492b1a4368936fac95c11e1f73 (127.0.252.254:36505): Couldn't send request to peer ef8906492b1a4368936fac95c11e1f73. Status: Network error: Client connection negotiation failed: client connection to 127.0.252.254:36505: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251024 19:48:09.090155 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeeaf256b1404653b690c712d9ef7a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.094926 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a9b28ff5674d5db8002b748e2733b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.100318 1617 tablet_service.cc:1505] Processing CreateTablet for tablet b4a66b7c89794807826bea29229a6d3a (DEFAULT_TABLE table=default.Table_a7d96771ce8440ebb201c4095b80c7c8 [id=09e7dbd329b74330b6dd071f40a79781]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.100756 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a66b7c89794807826bea29229a6d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.017916 1369 raft_consensus.cc:3060] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:09.027288 1336 tablet_service.cc:1505] Processing CreateTablet for tablet c1f4bb2e3a5b406fb5b37403667be5d8 (DEFAULT_TABLE table=default.Table_289bdf22b5834ac78a2b65cd33fac990 [id=0f2a7b323e8d495e95b198eee2e7319d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.038071 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 0143a393711446fdbe6c798efa0c8936 (DEFAULT_TABLE table=default.Table_5f895593b7d0404c909c32f99e24d5b3 [id=186dcdefc76a4e62b86cd9b9e12a17f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.041057 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 1e6a588092a44688bc96f16ac00e39f9 (DEFAULT_TABLE table=default.Table_114b0c5bec4643109970ce8f80add076 [id=e3e6a0009da04ec49dc6d0d5fa0e39e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.047712 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 2bb5f84c1f104b338be71cb55ae77929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:09.048117 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet e68997931ede4b6398570a7d08edaf4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:09.048309 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 95ae017a2e084ccabfa69c9a5befe0e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:09.073097 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e6a588092a44688bc96f16ac00e39f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:09.074115 1335 tablet_service.cc:1505] Processing CreateTablet for tablet cd69fd2bba504eb398e483d6215a8eb9 (DEFAULT_TABLE table=default.Table_d66f2a41a7a84e57a7abd1ac8cc172b6 [id=43a9f0a67da54813a02f1a9aa9247ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.079100 1334 tablet_service.cc:1505] Processing CreateTablet for tablet aeeaf256b1404653b690c712d9ef7a6d (DEFAULT_TABLE table=default.Table_5f488d12a77948e68c3b5c1152e003f7 [id=5ffa582758154e6694adf569169c096b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.085673 1343 tablet_service.cc:1505] Processing CreateTablet for tablet e5a9b28ff5674d5db8002b748e2733b5 (DEFAULT_TABLE table=default.Table_d0aa4354692e4f0c8b105c7ab7603590 [id=691cca934eed4899a85c7b8f147c74f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.104362 1628 tablet_service.cc:1505] Processing CreateTablet for tablet e68997931ede4b6398570a7d08edaf4d (DEFAULT_TABLE table=default.Table_afee33bb95144492a2c62b19a101c0e3 [id=4c20993642ce4cdd9908559d4405f0ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.104687 1621 tablet_service.cc:1505] Processing CreateTablet for tablet dd796d1bc1a74d1f99028fcc31b340cd (DEFAULT_TABLE table=default.Table_b25ae72792dd4a14adeb24e822c8b538 [id=52d3be6bf2bc41cb98803f0161c24465]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.104852 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 36ccb0cfba7a4a14b36d38897f4a9ef0 (DEFAULT_TABLE table=default.Table_8d2ad543f7a04e9e983cdaa997e94073 [id=2ebf9d4ca7114ee58d11cd9a745a09ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.104770 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68997931ede4b6398570a7d08edaf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.105273 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd796d1bc1a74d1f99028fcc31b340cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.105463 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36ccb0cfba7a4a14b36d38897f4a9ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.114887 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 23d4dd0b42d343cb9354ac698a8c761f with delete_type TABLET_DATA_DELETED (replaced by tablet fce7de9523fb4878a58aa8aef0c6b182) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:09.093056 2522 raft_consensus.cc:515] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:09.116379 1481 tablet_service.cc:1552] Processing DeleteTablet for tablet 2bb5f84c1f104b338be71cb55ae77929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:09.121927 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f4bb2e3a5b406fb5b37403667be5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.123250 2500 tablet.cc:1722] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:09.126332 2522 leader_election.cc:290] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:09.132825 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0143a393711446fdbe6c798efa0c8936. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.133140 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dfeadb429d564640af3d8d78fa50e64e" } new_table_name: "default.Table_082b885410b34b1cb2f154cdbd9700a8" modify_external_catalogs: true
I20251024 19:48:09.133229 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_35c6a6e931bc4596a02d717ead9865ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.133802 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35c6a6e931bc4596a02d717ead9865ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.134091 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_964aae2f085749baaf4f7d0ee3ebb1f4" } new_table_name: "default.Table_1f866642ce4a4ee885f169c4815bd1fb" modify_external_catalogs: true
I20251024 19:48:09.135857 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 2bb5f84c1f104b338be71cb55ae77929 (DEFAULT_TABLE table=default.Table_86b9f6987402431196ad8064631c34bc [id=8dd1c89095be4b70abe16d76832986cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.150497 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b905a1285caa407e9fb08e81639c9bc8" } new_table_name: "default.Table_2a9850f5d5dc499f87e87bdb0efcc93a" modify_external_catalogs: true
I20251024 19:48:09.150708 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_cfeecdf5f1bd4eb099bf97e5713a4d41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:09.150497 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_bab51dadce54487d96884e3ef240e684"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.151314 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bab51dadce54487d96884e3ef240e684 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.151446 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_733a79b22a954b088ad061fc232e3008"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.151901 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfeecdf5f1bd4eb099bf97e5713a4d41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:09.151980 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_733a79b22a954b088ad061fc232e3008 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.152287 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_434a6ef23bea4f3991418b6ed2d9f635"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.152724 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_434a6ef23bea4f3991418b6ed2d9f635 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.156981 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c7dbe9ff2c4869994c4f43d5676c30" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:09.160398 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_af63b6be7a7241feb3512fe1856876bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:09.160420 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_04c43324857b4aa7b8a4c4f6eb0769f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.161067 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af63b6be7a7241feb3512fe1856876bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:09.161067 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04c43324857b4aa7b8a4c4f6eb0769f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.162560 1611 tablet_service.cc:1505] Processing CreateTablet for tablet fc187c8f34594f4fb37d8afe0855263b (DEFAULT_TABLE table=default.Table_1db3c2ecc936440985b132c9db2c56e5 [id=92e7ec3ccc8840dcaa13422a78d3a15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.163105 1626 tablet_service.cc:1505] Processing CreateTablet for tablet 4d0e4ffcdf184c5bafd99bf721b35951 (DEFAULT_TABLE table=default.Table_1312dfdab750454a8f412ce6a66284ac [id=28dfd9c9c5cf470abbcfad936688bf94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.164264 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bb5f84c1f104b338be71cb55ae77929. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.168146 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d0e4ffcdf184c5bafd99bf721b35951. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.169207 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc187c8f34594f4fb37d8afe0855263b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.172701 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet e68997931ede4b6398570a7d08edaf4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:09.172974 1444 leader_election.cc:343] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:09.188126 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f4bb2e3a5b406fb5b37403667be5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.190299 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e6a588092a44688bc96f16ac00e39f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.193493 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0143a393711446fdbe6c798efa0c8936. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.195017 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd69fd2bba504eb398e483d6215a8eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.195760 1342 tablet_service.cc:1505] Processing CreateTablet for tablet b95e41141f5f4a5bbc39fb977d82252b (DEFAULT_TABLE table=default.Table_6a973600640545a6947cf458bf8ca233 [id=5fef72e759e74b6b8aad2516bc7e4911]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.196342 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 95ae017a2e084ccabfa69c9a5befe0e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:09.196645 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 6f7b6d15f25d4c2a905da6597ba8d98a (DEFAULT_TABLE table=default.Table_9e7fa6339ee24bab9d78fd7c318ed78d [id=b03b545fcd2f47168d4309e5a6c5ea2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.196990 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 2bb5f84c1f104b338be71cb55ae77929 (DEFAULT_TABLE table=default.Table_86b9f6987402431196ad8064631c34bc [id=8dd1c89095be4b70abe16d76832986cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.198268 1348 tablet_service.cc:1505] Processing CreateTablet for tablet dd796d1bc1a74d1f99028fcc31b340cd (DEFAULT_TABLE table=default.Table_b25ae72792dd4a14adeb24e822c8b538 [id=52d3be6bf2bc41cb98803f0161c24465]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.198601 1347 tablet_service.cc:1505] Processing CreateTablet for tablet eb6b7df6176e4bccacf7a9059cee5477 (DEFAULT_TABLE table=default.Table_f000460d66bd46a2bcae726de4f6fb1f [id=c5f0720af6a44d07a0980b7154ef3c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.199222 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 6f7b6d15f25d4c2a905da6597ba8d98a (DEFAULT_TABLE table=default.Table_9e7fa6339ee24bab9d78fd7c318ed78d [id=b03b545fcd2f47168d4309e5a6c5ea2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.199975 1629 tablet_service.cc:1505] Processing CreateTablet for tablet eb6b7df6176e4bccacf7a9059cee5477 (DEFAULT_TABLE table=default.Table_f000460d66bd46a2bcae726de4f6fb1f [id=c5f0720af6a44d07a0980b7154ef3c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.200537 1352 tablet_service.cc:1505] Processing CreateTablet for tablet b4a66b7c89794807826bea29229a6d3a (DEFAULT_TABLE table=default.Table_a7d96771ce8440ebb201c4095b80c7c8 [id=09e7dbd329b74330b6dd071f40a79781]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.201061 1350 tablet_service.cc:1505] Processing CreateTablet for tablet e68997931ede4b6398570a7d08edaf4d (DEFAULT_TABLE table=default.Table_afee33bb95144492a2c62b19a101c0e3 [id=4c20993642ce4cdd9908559d4405f0ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.201903 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b95e41141f5f4a5bbc39fb977d82252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.202766 1768 tablet_bootstrap.cc:492] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:09.204113 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a9b28ff5674d5db8002b748e2733b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.197957 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 36ccb0cfba7a4a14b36d38897f4a9ef0 (DEFAULT_TABLE table=default.Table_8d2ad543f7a04e9e983cdaa997e94073 [id=2ebf9d4ca7114ee58d11cd9a745a09ac]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:09.204252 2441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 19:48:09.205787 2441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 19:48:09.206357 2441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251024 19:48:09.210479 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeeaf256b1404653b690c712d9ef7a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.212623 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f7b6d15f25d4c2a905da6597ba8d98a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:09.222833 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 250f106664be4b5aaa330dbb4e43cb7b consensus state has no leader
I20251024 19:48:09.236827 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_56c54a42391a4397bdbcc0964ed0f9e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:09.236917 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7065c04cd8e6495180d009455573cde0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.237551 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56c54a42391a4397bdbcc0964ed0f9e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:09.237596 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7065c04cd8e6495180d009455573cde0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.239022 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d7e131f8c5f8432d8c00ffbf1857560e" } new_table_name: "default.Table_7610674d2b04400c8e953eb393f066e7" modify_external_catalogs: true
I20251024 19:48:09.241041 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 4d0e4ffcdf184c5bafd99bf721b35951 (DEFAULT_TABLE table=default.Table_1312dfdab750454a8f412ce6a66284ac [id=28dfd9c9c5cf470abbcfad936688bf94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.241315 1338 tablet_service.cc:1505] Processing CreateTablet for tablet fc187c8f34594f4fb37d8afe0855263b (DEFAULT_TABLE table=default.Table_1db3c2ecc936440985b132c9db2c56e5 [id=92e7ec3ccc8840dcaa13422a78d3a15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.241731 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bb5f84c1f104b338be71cb55ae77929. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.241968 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc187c8f34594f4fb37d8afe0855263b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:09.246588 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0797d14e9fd846788164747523df0e2c consensus state has no leader
I20251024 19:48:09.247751 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68997931ede4b6398570a7d08edaf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.249293 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d0e4ffcdf184c5bafd99bf721b35951. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.250149 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9c03ce6b892d44fcbff5c92f613bc37f" } modify_external_catalogs: true
I20251024 19:48:09.250341 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9c03ce6b892d44fcbff5c92f613bc37f" } modify_external_catalogs: true
I20251024 19:48:09.250919 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a66b7c89794807826bea29229a6d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.251507 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd796d1bc1a74d1f99028fcc31b340cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.251482 2419 raft_consensus.cc:493] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:09.251770 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb6b7df6176e4bccacf7a9059cee5477. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.251715 2419 raft_consensus.cc:515] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:09.252528 2419 leader_election.cc:290] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:09.256510 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36ccb0cfba7a4a14b36d38897f4a9ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.260933 1774 ts_tablet_manager.cc:1929] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:09.261061 1774 log.cc:1199] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/44ee03ede02c4bdab706fde910978332
I20251024 19:48:09.261507 1774 ts_tablet_manager.cc:1950] T 44ee03ede02c4bdab706fde910978332 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:09.261947 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 387) took 5346 ms. Trace:
I20251024 19:48:09.262046 1774 rpcz_store.cc:276] 1024 19:48:03.915492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:03.915803 (+ 311us) service_pool.cc:225] Handling call
1024 19:48:08.849799 (+4933996us) ts_tablet_manager.cc:1151] Deleting tablet 44ee03ede02c4bdab706fde910978332
1024 19:48:08.849814 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:08.865749 (+ 15935us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.260833 (+395084us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.261927 (+ 1094us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4933789}
I20251024 19:48:09.262470 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 397) took 5020 ms. Trace:
I20251024 19:48:09.262557 1774 rpcz_store.cc:276] 1024 19:48:04.241771 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.243383 (+ 1612us) service_pool.cc:225] Handling call
1024 19:48:09.262342 (+5018959us) ts_tablet_manager.cc:1151] Deleting tablet af43ffe097e14c4faffed95e207d0893
1024 19:48:09.262352 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.262453 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5018670}
I20251024 19:48:09.262885 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 398) took 5020 ms. Trace:
I20251024 19:48:09.262965 1774 rpcz_store.cc:276] 1024 19:48:04.242298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.243711 (+ 1413us) service_pool.cc:225] Handling call
1024 19:48:09.262782 (+5019071us) ts_tablet_manager.cc:1151] Deleting tablet d38154d3ffd34f4b8d7305b1c23a4fae
1024 19:48:09.262791 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.262869 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5018927}
I20251024 19:48:09.263231 1774 tablet_replica.cc:333] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:09.263411 1774 raft_consensus.cc:2243] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:09.263563 1774 raft_consensus.cc:2272] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:09.264092 1774 ts_tablet_manager.cc:1916] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:09.264781 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 44ee03ede02c4bdab706fde910978332 (table default.Table_5f1d0b12b2034f5185f21c87157cf051 [id=46806f1d4f434926bc23214f0e3605d8]) successfully deleted
W20251024 19:48:09.265107 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet af43ffe097e14c4faffed95e207d0893 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af43ffe097e14c4faffed95e207d0893 already in progress: creating tablet
W20251024 19:48:09.265398 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet d38154d3ffd34f4b8d7305b1c23a4fae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d38154d3ffd34f4b8d7305b1c23a4fae already in progress: creating tablet
W20251024 19:48:09.270134 2441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 19:48:09.270249 2441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 19:48:09.270334 2441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20251024 19:48:09.270313 1768 tablet_bootstrap.cc:654] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:09.270397 2441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 19:48:09.275250 1774 ts_tablet_manager.cc:1929] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:09.275400 1774 log.cc:1199] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/4db17d9aff6542b6b94c09f872a4cceb
I20251024 19:48:09.275696 1774 ts_tablet_manager.cc:1950] T 4db17d9aff6542b6b94c09f872a4cceb P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:09.276036 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 400) took 4990 ms. Trace:
I20251024 19:48:09.276119 1774 rpcz_store.cc:276] 1024 19:48:04.285155 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.285210 (+ 55us) service_pool.cc:225] Handling call
1024 19:48:09.263186 (+4977976us) ts_tablet_manager.cc:1151] Deleting tablet 4db17d9aff6542b6b94c09f872a4cceb
1024 19:48:09.263196 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.271453 (+ 8257us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.275144 (+ 3691us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.276017 (+ 873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4977714}
I20251024 19:48:09.276749 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 4db17d9aff6542b6b94c09f872a4cceb (table default.Table_ef3fdce598e74165bddac418dc92c20e [id=e4ee1d33b7e24827aa24950ea40465e4]) successfully deleted
I20251024 19:48:09.276542 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 399) took 4993 ms. Trace:
I20251024 19:48:09.278369 1774 rpcz_store.cc:276] 1024 19:48:04.283572 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.285517 (+ 1945us) service_pool.cc:225] Handling call
1024 19:48:09.276418 (+4990901us) ts_tablet_manager.cc:1151] Deleting tablet d1d26e02ad2945b39cf27738ed5b7cf4
1024 19:48:09.276428 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.276527 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4990678}
I20251024 19:48:09.278810 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 402) took 4957 ms. Trace:
W20251024 19:48:09.278915 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet d1d26e02ad2945b39cf27738ed5b7cf4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1d26e02ad2945b39cf27738ed5b7cf4 already in progress: creating tablet
I20251024 19:48:09.278898 1774 rpcz_store.cc:276] 1024 19:48:04.321048 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.329652 (+ 8604us) service_pool.cc:225] Handling call
1024 19:48:09.278670 (+4949018us) ts_tablet_manager.cc:1151] Deleting tablet ad81d80697984ff1834a7d795e0f4b9b
1024 19:48:09.278680 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.278791 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4948271}
I20251024 19:48:09.279224 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 401) took 4958 ms. Trace:
I20251024 19:48:09.279300 1774 rpcz_store.cc:276] 1024 19:48:04.320673 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.331554 (+ 10881us) service_pool.cc:225] Handling call
1024 19:48:09.279126 (+4947572us) ts_tablet_manager.cc:1151] Deleting tablet 4243f11bf2b34b30b10766c19e2c6409
1024 19:48:09.279135 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.279209 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4946680}
I20251024 19:48:09.279631 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 406) took 4608 ms. Trace:
W20251024 19:48:09.279634 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4243f11bf2b34b30b10766c19e2c6409 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4243f11bf2b34b30b10766c19e2c6409 already in progress: creating tablet
I20251024 19:48:09.279723 1774 rpcz_store.cc:276] 1024 19:48:04.671641 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.672330 (+ 689us) service_pool.cc:225] Handling call
1024 19:48:09.279503 (+4607173us) ts_tablet_manager.cc:1151] Deleting tablet 163ac050c8574615aef5061f563cfc0a
1024 19:48:09.279514 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.279612 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4606846}
I20251024 19:48:09.280037 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 408) took 4580 ms. Trace:
W20251024 19:48:09.280169 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 163ac050c8574615aef5061f563cfc0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 163ac050c8574615aef5061f563cfc0a already in progress: creating tablet
I20251024 19:48:09.280128 1774 rpcz_store.cc:276] 1024 19:48:04.699621 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.699689 (+ 68us) service_pool.cc:225] Handling call
1024 19:48:09.279939 (+4580250us) ts_tablet_manager.cc:1151] Deleting tablet f83b9290f00145a780318a43249eec12
1024 19:48:09.279947 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.280023 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4580000}
I20251024 19:48:09.280450 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 409) took 4550 ms. Trace:
I20251024 19:48:09.280529 1774 rpcz_store.cc:276] 1024 19:48:04.730139 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.730199 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:09.280360 (+4550161us) ts_tablet_manager.cc:1151] Deleting tablet 9b89a90bb4ca447fbaf34fe39001c261
1024 19:48:09.280369 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.280434 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4549955}
W20251024 19:48:09.280653 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet f83b9290f00145a780318a43249eec12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f83b9290f00145a780318a43249eec12 already in progress: creating tablet
I20251024 19:48:09.280864 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 413) took 4499 ms. Trace:
I20251024 19:48:09.280952 1774 rpcz_store.cc:276] 1024 19:48:04.781201 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.781257 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:09.280767 (+4499510us) ts_tablet_manager.cc:1151] Deleting tablet 3cf4596b333244cc9e9e35a128721e4a
1024 19:48:09.280776 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.280849 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4499252}
W20251024 19:48:09.281169 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9b89a90bb4ca447fbaf34fe39001c261 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b89a90bb4ca447fbaf34fe39001c261 already in progress: creating tablet
I20251024 19:48:09.281248 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 418) took 4447 ms. Trace:
I20251024 19:48:09.281351 1774 rpcz_store.cc:276] 1024 19:48:04.834258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.834593 (+ 335us) service_pool.cc:225] Handling call
1024 19:48:09.281149 (+4446556us) ts_tablet_manager.cc:1151] Deleting tablet ebcc09d8e029451d8fb6b6601ad16087
1024 19:48:09.281159 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.281231 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4446313}
W20251024 19:48:09.281472 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3cf4596b333244cc9e9e35a128721e4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cf4596b333244cc9e9e35a128721e4a already in progress: creating tablet
I20251024 19:48:09.281680 1774 tablet_replica.cc:333] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:09.281896 1774 raft_consensus.cc:2243] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:09.282054 1774 pending_rounds.cc:70] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
W20251024 19:48:09.285053 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet ebcc09d8e029451d8fb6b6601ad16087 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebcc09d8e029451d8fb6b6601ad16087 already in progress: creating tablet
I20251024 19:48:09.282097 1774 pending_rounds.cc:77] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7214429332057751552 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "c04082bd1b444b468c177bef239701c6" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 2 new_table_name: "default.Table_24200cd6f19d460cae963afb9b7f8724" dest_uuid: "af3952de050140d383f6b0de7666fa97" new_extra_config { } }
I20251024 19:48:09.285876 1774 raft_consensus.cc:2272] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:09.286486 1774 ts_tablet_manager.cc:1916] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:09.293177 1774 ts_tablet_manager.cc:1929] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:09.293327 1774 log.cc:1199] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/c04082bd1b444b468c177bef239701c6
I20251024 19:48:09.293769 1774 ts_tablet_manager.cc:1950] T c04082bd1b444b468c177bef239701c6 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:09.294121 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 419) took 4459 ms. Trace:
I20251024 19:48:09.294207 1774 rpcz_store.cc:276] 1024 19:48:04.834388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.834879 (+ 491us) service_pool.cc:225] Handling call
1024 19:48:09.281631 (+4446752us) ts_tablet_manager.cc:1151] Deleting tablet c04082bd1b444b468c177bef239701c6
1024 19:48:09.281641 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.290003 (+ 8362us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.293068 (+ 3065us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.294101 (+ 1033us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4446549}
I20251024 19:48:09.294636 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 420) took 4460 ms. Trace:
I20251024 19:48:09.294716 1774 rpcz_store.cc:276] 1024 19:48:04.834537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.835110 (+ 573us) service_pool.cc:225] Handling call
1024 19:48:09.294510 (+4459400us) ts_tablet_manager.cc:1151] Deleting tablet 6ec5c7bf36874423b4790c6169da8bd0
1024 19:48:09.294519 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.294621 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4459244}
I20251024 19:48:09.295019 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 421) took 4335 ms. Trace:
I20251024 19:48:09.295089 1774 rpcz_store.cc:276] 1024 19:48:04.959944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:04.960122 (+ 178us) service_pool.cc:225] Handling call
1024 19:48:09.294933 (+4334811us) ts_tablet_manager.cc:1151] Deleting tablet 6424b2347ef642feb38a9596f2d2be1d
1024 19:48:09.294941 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.295008 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4334576}
I20251024 19:48:09.295373 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 422) took 4272 ms. Trace:
I20251024 19:48:09.295447 1774 rpcz_store.cc:276] 1024 19:48:05.022815 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.074421 (+ 51606us) service_pool.cc:225] Handling call
1024 19:48:09.295281 (+4220860us) ts_tablet_manager.cc:1151] Deleting tablet ab488e4971874a16b2e052c84c250777
1024 19:48:09.295288 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.295359 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4220496}
I20251024 19:48:09.295676 1774 tablet_replica.cc:333] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:09.295872 1774 raft_consensus.cc:2243] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:09.296068 1774 raft_consensus.cc:2272] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:09.296204 2441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.252.212:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.252.254:36505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.252.254
--webserver_port=42051
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 24 Oct 2025 19:43:25 UTC on 24a791456cd2
build id 8700
ASAN enabled
I20251024 19:48:09.296831 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet c04082bd1b444b468c177bef239701c6 (table default.Table_24200cd6f19d460cae963afb9b7f8724 [id=a2789faf4abe4c09b0a7515d941831a6]) successfully deleted
W20251024 19:48:09.297292 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 6424b2347ef642feb38a9596f2d2be1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6424b2347ef642feb38a9596f2d2be1d already in progress: creating tablet
I20251024 19:48:09.299469 2441 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20251024 19:48:09.301357 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet ab488e4971874a16b2e052c84c250777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab488e4971874a16b2e052c84c250777 already in progress: creating tablet
I20251024 19:48:09.303786 1774 ts_tablet_manager.cc:1916] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:09.304764 2441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251024 19:48:09.322691 1766 tablet_bootstrap.cc:654] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:09.329209 2562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 19:48:09.342048 2563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 19:48:09.344153 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 1e6a588092a44688bc96f16ac00e39f9 (DEFAULT_TABLE table=default.Table_114b0c5bec4643109970ce8f80add076 [id=e3e6a0009da04ec49dc6d0d5fa0e39e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.344800 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e6a588092a44688bc96f16ac00e39f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.345882 1474 tablet_service.cc:1505] Processing CreateTablet for tablet cd69fd2bba504eb398e483d6215a8eb9 (DEFAULT_TABLE table=default.Table_d66f2a41a7a84e57a7abd1ac8cc172b6 [id=43a9f0a67da54813a02f1a9aa9247ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.346453 1768 tablet_bootstrap.cc:492] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:09.346611 1768 ts_tablet_manager.cc:1403] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.144s user 0.005s sys 0.000s
I20251024 19:48:09.351946 1774 ts_tablet_manager.cc:1929] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:09.352054 1774 log.cc:1199] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/dc1b6f0a377945089c50012f74d38564
I20251024 19:48:09.352388 1774 ts_tablet_manager.cc:1950] T dc1b6f0a377945089c50012f74d38564 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:09.352731 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd69fd2bba504eb398e483d6215a8eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.352746 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 423) took 4050 ms. Trace:
I20251024 19:48:09.352810 1774 rpcz_store.cc:276] 1024 19:48:05.302452 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.302521 (+ 69us) service_pool.cc:225] Handling call
1024 19:48:09.295628 (+3993107us) ts_tablet_manager.cc:1151] Deleting tablet dc1b6f0a377945089c50012f74d38564
1024 19:48:09.295637 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.310106 (+ 14469us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.351844 (+ 41738us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.352729 (+ 885us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3992856}
I20251024 19:48:09.353093 1774 tablet_replica.cc:333] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:09.353286 1774 raft_consensus.cc:2243] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:09.353454 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e6a588092a44688bc96f16ac00e39f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:09.354307 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet dc1b6f0a377945089c50012f74d38564 (table default.Table_52ec55c23c394294a68850ab6b33aabe [id=fb0e7cab20f44aaebdb09862bd193f41]) successfully deleted
I20251024 19:48:09.355100 1489 tablet_service.cc:1505] Processing CreateTablet for tablet e68997931ede4b6398570a7d08edaf4d (DEFAULT_TABLE table=default.Table_afee33bb95144492a2c62b19a101c0e3 [id=4c20993642ce4cdd9908559d4405f0ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.355551 1481 tablet_service.cc:1505] Processing CreateTablet for tablet 2bb5f84c1f104b338be71cb55ae77929 (DEFAULT_TABLE table=default.Table_86b9f6987402431196ad8064631c34bc [id=8dd1c89095be4b70abe16d76832986cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.357482 1774 raft_consensus.cc:2272] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:09.357791 1481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bb5f84c1f104b338be71cb55ae77929. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.358125 1774 ts_tablet_manager.cc:1916] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:09.356985 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca0527af718f494a8bbcc66e7b325bed" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:09.359556 1478 tablet_service.cc:1505] Processing CreateTablet for tablet b95e41141f5f4a5bbc39fb977d82252b (DEFAULT_TABLE table=default.Table_6a973600640545a6947cf458bf8ca233 [id=5fef72e759e74b6b8aad2516bc7e4911]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.359907 1478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b95e41141f5f4a5bbc39fb977d82252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.360983 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c7dbe9ff2c4869994c4f43d5676c30" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:09.362296 1485 tablet_service.cc:1505] Processing CreateTablet for tablet fc187c8f34594f4fb37d8afe0855263b (DEFAULT_TABLE table=default.Table_1db3c2ecc936440985b132c9db2c56e5 [id=92e7ec3ccc8840dcaa13422a78d3a15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.362840 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc187c8f34594f4fb37d8afe0855263b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.364020 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 36ccb0cfba7a4a14b36d38897f4a9ef0 (DEFAULT_TABLE table=default.Table_8d2ad543f7a04e9e983cdaa997e94073 [id=2ebf9d4ca7114ee58d11cd9a745a09ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.364456 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36ccb0cfba7a4a14b36d38897f4a9ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.365200 1486 tablet_service.cc:1505] Processing CreateTablet for tablet dd796d1bc1a74d1f99028fcc31b340cd (DEFAULT_TABLE table=default.Table_b25ae72792dd4a14adeb24e822c8b538 [id=52d3be6bf2bc41cb98803f0161c24465]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.365435 1774 ts_tablet_manager.cc:1929] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:09.366070 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd796d1bc1a74d1f99028fcc31b340cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.366768 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 6f7b6d15f25d4c2a905da6597ba8d98a (DEFAULT_TABLE table=default.Table_9e7fa6339ee24bab9d78fd7c318ed78d [id=b03b545fcd2f47168d4309e5a6c5ea2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.366936 1488 tablet_service.cc:1505] Processing CreateTablet for tablet e5a9b28ff5674d5db8002b748e2733b5 (DEFAULT_TABLE table=default.Table_d0aa4354692e4f0c8b105c7ab7603590 [id=691cca934eed4899a85c7b8f147c74f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.367194 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 4d0e4ffcdf184c5bafd99bf721b35951 (DEFAULT_TABLE table=default.Table_1312dfdab750454a8f412ce6a66284ac [id=28dfd9c9c5cf470abbcfad936688bf94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.367743 1509 raft_consensus.cc:2468] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:09.368064 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d0e4ffcdf184c5bafd99bf721b35951. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.369109 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 59c7dbe9ff2c4869994c4f43d5676c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:09.369346 1480 tablet_service.cc:1505] Processing CreateTablet for tablet b4a66b7c89794807826bea29229a6d3a (DEFAULT_TABLE table=default.Table_a7d96771ce8440ebb201c4095b80c7c8 [id=09e7dbd329b74330b6dd071f40a79781]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.369772 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a66b7c89794807826bea29229a6d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.370486 1477 tablet_service.cc:1505] Processing CreateTablet for tablet aeeaf256b1404653b690c712d9ef7a6d (DEFAULT_TABLE table=default.Table_5f488d12a77948e68c3b5c1152e003f7 [id=5ffa582758154e6694adf569169c096b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.370923 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeeaf256b1404653b690c712d9ef7a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.371595 1475 tablet_service.cc:1505] Processing CreateTablet for tablet eb6b7df6176e4bccacf7a9059cee5477 (DEFAULT_TABLE table=default.Table_f000460d66bd46a2bcae726de4f6fb1f [id=c5f0720af6a44d07a0980b7154ef3c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.371980 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb6b7df6176e4bccacf7a9059cee5477. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.375623 1774 log.cc:1199] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/8a1f1892bd3b4776adc7c2519fcdf411
I20251024 19:48:09.378603 2419 consensus_queue.cc:1048] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:09.380184 1618 tablet_service.cc:1505] Processing CreateTablet for tablet b95e41141f5f4a5bbc39fb977d82252b (DEFAULT_TABLE table=default.Table_6a973600640545a6947cf458bf8ca233 [id=5fef72e759e74b6b8aad2516bc7e4911]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:09.388221 1444 consensus_peers.cc:597] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aebf76d4a66947ed9756643740418798. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:09.400431 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 593f5600e4274fce97e7bd33f025569f consensus state has no leader
I20251024 19:48:09.400480 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_63e00ebf66c2426a96902b5443eaa366" } new_table_name: "default.Table_319540d9d2544798ac89ac3c8406f4ce" modify_external_catalogs: true
I20251024 19:48:09.401183 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f11b271966af455798227788699910b6" } modify_external_catalogs: true
W20251024 19:48:09.401568 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 769f576810ab49368eb0fc7837b7da58 consensus state has no leader
I20251024 19:48:09.401953 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f5d35eb71b3948ffbc2a9c015823e67f" } modify_external_catalogs: true
I20251024 19:48:09.402015 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b82f8da2443f4f0f91dd2b762983abac" } modify_external_catalogs: true
I20251024 19:48:09.402123 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f5d35eb71b3948ffbc2a9c015823e67f" } modify_external_catalogs: true
I20251024 19:48:09.402577 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1606f259eb4f47678e56892bf267bf77" } modify_external_catalogs: true
I20251024 19:48:09.402757 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1606f259eb4f47678e56892bf267bf77" } modify_external_catalogs: true
I20251024 19:48:09.402931 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1dbcae61809349c59e959bff6c5cdf24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.403522 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dbcae61809349c59e959bff6c5cdf24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.403811 1768 raft_consensus.cc:359] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:09.403960 1774 ts_tablet_manager.cc:1950] T 8a1f1892bd3b4776adc7c2519fcdf411 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:09.405436 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 426) took 3922 ms. Trace:
I20251024 19:48:09.403986 1768 raft_consensus.cc:385] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:09.406692 1768 raft_consensus.cc:740] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:09.403528 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet 4db17d9aff6542b6b94c09f872a4cceb from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:09.406885 1768 consensus_queue.cc:260] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:09.402132 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b82f8da2443f4f0f91dd2b762983abac" } modify_external_catalogs: true
I20251024 19:48:09.407361 1768 ts_tablet_manager.cc:1434] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.061s user 0.002s sys 0.000s
I20251024 19:48:09.405776 1774 rpcz_store.cc:276] 1024 19:48:05.482522 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.482588 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:09.353053 (+3870465us) ts_tablet_manager.cc:1151] Deleting tablet 8a1f1892bd3b4776adc7c2519fcdf411
1024 19:48:09.353062 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.362219 (+ 9157us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.365357 (+ 3138us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.405311 (+ 39954us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3870167}
I20251024 19:48:09.407645 1768 tablet_bootstrap.cc:492] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:09.410358 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 430) took 3877 ms. Trace:
I20251024 19:48:09.419015 1586 leader_election.cc:304] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:09.419925 2556 tablet.cc:1722] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:09.401608 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f11b271966af455798227788699910b6" } modify_external_catalogs: true
I20251024 19:48:09.424181 1188 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet ce0ef4443e574846a9f48fef3b893352 (table default.Table_f203b363de4c47fca0a0599140d18279 [id=ec57efb4d37e44f1b68045f58bc11c45]). Expected version 1 got 0
I20251024 19:48:09.432276 2501 raft_consensus.cc:2804] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:09.432524 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c570270b7d7b4f5091c34e0550dd46ef" } new_table_name: "default.Table_0767e8424414437b9dd0a83e1ce6ad4d" modify_external_catalogs: true
I20251024 19:48:09.433782 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 8a1f1892bd3b4776adc7c2519fcdf411 (table default.Table_46b7ffb5bf6f45f2be4d1b242f43bfc9 [id=0cd6320a10064e9a8556836dfdb2a1dc]) successfully deleted
I20251024 19:48:09.433843 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d10102cfa0fa4eb9b806bacdd5066807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.434502 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d10102cfa0fa4eb9b806bacdd5066807 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.439874 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a6498b82a207414dafe80255fff08e8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.440526 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6498b82a207414dafe80255fff08e8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.441221 2501 raft_consensus.cc:493] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:09.443320 2501 raft_consensus.cc:3060] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:09.444651 1612 tablet_service.cc:1505] Processing CreateTablet for tablet d2c92975cf66441f9ac90c19772d3db3 (DEFAULT_TABLE table=default.Table_1606f259eb4f47678e56892bf267bf77 [id=342122e08a644b7c8758b89d588e3d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.446331 1625 tablet_service.cc:1505] Processing CreateTablet for tablet f9d09a94d1ea4af19b3da5155b3c4aad (DEFAULT_TABLE table=default.Table_7837f9d39d7e4658a8945b4ddd2cf236 [id=1370e40179c5440185786cfae2bbbd7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.447458 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 57fed826b4b64edba48c5fffaef1e031 (DEFAULT_TABLE table=default.Table_6501382adb09425e8fa508d81fe78a24 [id=c5a5612d649d415dac70547f7a1e7f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.447817 1630 tablet_service.cc:1505] Processing CreateTablet for tablet f4e812c43ade4831af50825474976c8e (DEFAULT_TABLE table=default.Table_bd37f8f763d147b2b7656b63f72fd749 [id=91696606f250441eaeade9c5cad1cc5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.448894 1767 tablet_bootstrap.cc:492] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:09.449059 1767 ts_tablet_manager.cc:1403] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.555s user 0.030s sys 0.054s
I20251024 19:48:09.450704 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 8359a8d5bf544cca9b2407a650f3c9b1 (DEFAULT_TABLE table=default.Table_6eab640e6ecf469aa77cd34cbfa7b361 [id=e5179c71774c47938da28d18b02cc812]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.451156 1767 raft_consensus.cc:359] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:09.451321 1767 raft_consensus.cc:385] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:09.450619 1620 tablet_service.cc:1505] Processing CreateTablet for tablet cbd321c059e34834a6257fab5e6d14bb (DEFAULT_TABLE table=default.Table_345aac1447b347f99accafba7777859a [id=0b6b51df83d54f6986b2b53791d2083b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:09.452920 1586 consensus_peers.cc:597] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:09.454071 1613 tablet_service.cc:1505] Processing CreateTablet for tablet ce5b2cfcc8294b3ea8b89334786006f8 (DEFAULT_TABLE table=default.Table_f57b3f3656d347a3aee8146fd991ca1e [id=5cf6d30527544e698e0fd68ad20c66eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.454301 1627 tablet_service.cc:1505] Processing CreateTablet for tablet fce7de9523fb4878a58aa8aef0c6b182 (DEFAULT_TABLE table=default.Table_628b2afa9401426980be0eae5dffd917 [id=5b96ec1514ae41e3a0103e4b381aab61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.454973 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 9377725db312439e8a42dbd46da84a9d (DEFAULT_TABLE table=default.Table_f186552a977c4a1e8002ece18abd9d07 [id=2d70ca56de444b8bbc638451232f591b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.456575 1622 tablet_service.cc:1505] Processing CreateTablet for tablet a7ffb227901f49b99fb57c5c6e138c8c (DEFAULT_TABLE table=default.Table_9ebd164c5e924a7eaff697af91c4c1a5 [id=6a388834945a480dabf5191d604d509f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.457648 2419 raft_consensus.cc:493] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:09.457814 2419 raft_consensus.cc:515] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:09.459010 2419 leader_election.cc:290] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:09.459725 1767 raft_consensus.cc:740] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
W20251024 19:48:09.466817 2570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 19:48:09.494522 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f7b6d15f25d4c2a905da6597ba8d98a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.411523 1774 rpcz_store.cc:276] 1024 19:48:05.532741 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.532810 (+ 69us) service_pool.cc:225] Handling call
1024 19:48:09.408795 (+3875985us) ts_tablet_manager.cc:1151] Deleting tablet 15de0d8aa9e545c09233186bbe361b97
1024 19:48:09.408932 (+ 137us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.410157 (+ 1225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3875310}
W20251024 19:48:09.503753 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 15de0d8aa9e545c09233186bbe361b97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15de0d8aa9e545c09233186bbe361b97 already in progress: creating tablet
I20251024 19:48:09.503822 1774 tablet_replica.cc:333] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:09.419504 1768 tablet_bootstrap.cc:654] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:09.504393 1774 raft_consensus.cc:2243] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:09.505151 1774 raft_consensus.cc:2272] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:09.431013 1351 tablet_service.cc:1505] Processing CreateTablet for tablet a7ffb227901f49b99fb57c5c6e138c8c (DEFAULT_TABLE table=default.Table_9ebd164c5e924a7eaff697af91c4c1a5 [id=6a388834945a480dabf5191d604d509f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.506551 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7ffb227901f49b99fb57c5c6e138c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.507145 1767 consensus_queue.cc:260] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:09.508637 2501 raft_consensus.cc:515] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:09.431330 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 9377725db312439e8a42dbd46da84a9d (DEFAULT_TABLE table=default.Table_f186552a977c4a1e8002ece18abd9d07 [id=2d70ca56de444b8bbc638451232f591b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.509707 1767 ts_tablet_manager.cc:1434] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.059s user 0.002s sys 0.000s
I20251024 19:48:09.509969 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9377725db312439e8a42dbd46da84a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.510010 2501 leader_election.cc:290] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:09.431591 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 8359a8d5bf544cca9b2407a650f3c9b1 (DEFAULT_TABLE table=default.Table_6eab640e6ecf469aa77cd34cbfa7b361 [id=e5179c71774c47938da28d18b02cc812]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.513947 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8359a8d5bf544cca9b2407a650f3c9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.514086 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd69fd2bba504eb398e483d6215a8eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.514999 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b95e41141f5f4a5bbc39fb977d82252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.431885 1346 tablet_service.cc:1505] Processing CreateTablet for tablet fce7de9523fb4878a58aa8aef0c6b182 (DEFAULT_TABLE table=default.Table_628b2afa9401426980be0eae5dffd917 [id=5b96ec1514ae41e3a0103e4b381aab61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.517765 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f7b6d15f25d4c2a905da6597ba8d98a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.517958 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fce7de9523fb4878a58aa8aef0c6b182. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.518314 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb6b7df6176e4bccacf7a9059cee5477. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.519405 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c92975cf66441f9ac90c19772d3db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.520843 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9d09a94d1ea4af19b3da5155b3c4aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.436053 1338 tablet_service.cc:1505] Processing CreateTablet for tablet ce5b2cfcc8294b3ea8b89334786006f8 (DEFAULT_TABLE table=default.Table_f57b3f3656d347a3aee8146fd991ca1e [id=5cf6d30527544e698e0fd68ad20c66eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.521565 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57fed826b4b64edba48c5fffaef1e031. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.523187 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce5b2cfcc8294b3ea8b89334786006f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.523932 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4e812c43ade4831af50825474976c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.525733 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fce7de9523fb4878a58aa8aef0c6b182. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.529979 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7ffb227901f49b99fb57c5c6e138c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.532650 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd321c059e34834a6257fab5e6d14bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.533223 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce5b2cfcc8294b3ea8b89334786006f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.436291 1348 tablet_service.cc:1505] Processing CreateTablet for tablet f4e812c43ade4831af50825474976c8e (DEFAULT_TABLE table=default.Table_bd37f8f763d147b2b7656b63f72fd749 [id=91696606f250441eaeade9c5cad1cc5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.537029 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4e812c43ade4831af50825474976c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.542362 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9377725db312439e8a42dbd46da84a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.436543 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 57fed826b4b64edba48c5fffaef1e031 (DEFAULT_TABLE table=default.Table_6501382adb09425e8fa508d81fe78a24 [id=c5a5612d649d415dac70547f7a1e7f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.543154 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57fed826b4b64edba48c5fffaef1e031. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.436801 1343 tablet_service.cc:1505] Processing CreateTablet for tablet cbd321c059e34834a6257fab5e6d14bb (DEFAULT_TABLE table=default.Table_345aac1447b347f99accafba7777859a [id=0b6b51df83d54f6986b2b53791d2083b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.546612 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd321c059e34834a6257fab5e6d14bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.437038 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 64d8da45c6364f1e9ddc79307904587c (DEFAULT_TABLE table=default.Table_379c43ee27bd4c74ae00adf4df216fa5 [id=9d26ddec659e4b9ba0530c10dec44397]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.550536 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d8da45c6364f1e9ddc79307904587c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.437278 1345 tablet_service.cc:1505] Processing CreateTablet for tablet f9d09a94d1ea4af19b3da5155b3c4aad (DEFAULT_TABLE table=default.Table_7837f9d39d7e4658a8945b4ddd2cf236 [id=1370e40179c5440185786cfae2bbbd7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.554200 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9d09a94d1ea4af19b3da5155b3c4aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.449615 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca0527af718f494a8bbcc66e7b325bed" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:09.460443 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73ffe342f541447d8519ee24bc98a0f3" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:09.432137 1352 tablet_service.cc:1505] Processing CreateTablet for tablet d2c92975cf66441f9ac90c19772d3db3 (DEFAULT_TABLE table=default.Table_1606f259eb4f47678e56892bf267bf77 [id=342122e08a644b7c8758b89d588e3d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.557540 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c92975cf66441f9ac90c19772d3db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.560190 2572 tablet.cc:1722] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:09.568481 1774 ts_tablet_manager.cc:1916] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:09.587213 2441 server_base.cc:1047] running on GCE node
I20251024 19:48:09.588887 2441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 19:48:09.600800 1767 tablet_bootstrap.cc:492] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:09.601477 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 59956dad06064024ab171631ef6c1dca
I20251024 19:48:09.603309 1186 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet ce0ef4443e574846a9f48fef3b893352 (table default.Table_f203b363de4c47fca0a0599140d18279 [id=ec57efb4d37e44f1b68045f58bc11c45]). Expected version 1 got 0
I20251024 19:48:09.604023 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet ce0ef4443e574846a9f48fef3b893352 (table default.Table_f203b363de4c47fca0a0599140d18279 [id=ec57efb4d37e44f1b68045f58bc11c45]). Expected version 1 got 0
I20251024 19:48:09.604267 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet ad055dc3387b414bb55d09b9683ba17e (table default.Table_cde3670bf0d04dc0b2049193bcee10d4 [id=c3de6937e93a48dcb88cabee59a09d32]). Expected version 1 got 0
I20251024 19:48:09.604243 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_581140b54cae4ac09f8e9c1b940e4241" } new_table_name: "default.Table_ff301124b8ba431cb56766ff580f43ba" modify_external_catalogs: true
I20251024 19:48:09.605273 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet d2c92975cf66441f9ac90c19772d3db3
W20251024 19:48:09.605986 1179 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 0e3a8d735a1741cabcabd64964f172fd (table default.Table_498bb6a3b91644e8831918efd611298d [id=18dad2fe2ca64f53bfe66b07b43aae76], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0e3a8d735a1741cabcabd64964f172fd consensus state has no leader
I20251024 19:48:09.607326 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 64499c8734d34c66b0b52a2cd3ef0673
W20251024 19:48:09.610249 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68ca2dd438d94bf0b5f1fc69f0a94517 consensus state has no leader
I20251024 19:48:09.611039 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fc2a1aeddc444f578693d922323862f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.611685 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc2a1aeddc444f578693d922323862f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.611605 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d101352faa2b4f4b9b983143a5142043"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:09.612946 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_715723e6de6a4ad0861818fa05311861" } new_table_name: "default.Table_18e7285b7df44399ab3be1dfc86eb91f" modify_external_catalogs: true
W20251024 19:48:09.613391 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bfe5aa5bd9844560874613d445ba5cc1 consensus state has no leader
I20251024 19:48:09.613936 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2087e295ad4b4daa8eeebd49b43e08c8
I20251024 19:48:09.614001 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fa23a3c70c3247f9ac84e69f79a41640"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.614416 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d101352faa2b4f4b9b983143a5142043 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:09.614708 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa23a3c70c3247f9ac84e69f79a41640 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.622776 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fb1d11002e79489c89a631ed3d11a5c3" } new_table_name: "default.Table_42c3a4a03db14d0684721ce6af6960c2" modify_external_catalogs: true
I20251024 19:48:09.624329 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 6f89a3bbeb9b4d57a49078729110d951
I20251024 19:48:09.630793 2548 raft_consensus.cc:493] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:09.630911 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 64d8da45c6364f1e9ddc79307904587c (DEFAULT_TABLE table=default.Table_379c43ee27bd4c74ae00adf4df216fa5 [id=9d26ddec659e4b9ba0530c10dec44397]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.630973 2548 raft_consensus.cc:515] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:09.631800 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9fee2e99d42e4cccae5d6e4825802713"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:09.632463 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7b543dc2852946a78c82d145655196ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.633100 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b543dc2852946a78c82d145655196ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.633647 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b1d45a27e5bd42e4a24d1a89f49f9221"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.634191 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1d45a27e5bd42e4a24d1a89f49f9221 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.635452 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73ffe342f541447d8519ee24bc98a0f3" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:09.635608 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3abf70710bce4728bfecd4a89d31dea6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.636082 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3abf70710bce4728bfecd4a89d31dea6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:09.633096 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fee2e99d42e4cccae5d6e4825802713 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.641613 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68997931ede4b6398570a7d08edaf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.646564 2441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 19:48:09.647500 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca0527af718f494a8bbcc66e7b325bed" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:09.647696 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet d2c92975cf66441f9ac90c19772d3db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:09.649597 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 59956dad06064024ab171631ef6c1dca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:09.649959 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 2087e295ad4b4daa8eeebd49b43e08c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:09.650137 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f89a3bbeb9b4d57a49078729110d951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:09.650269 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet 64499c8734d34c66b0b52a2cd3ef0673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:09.650514 2441 hybrid_clock.cc:648] HybridClock initialized: now 1761335289650415 us; error 83 us; skew 500 ppm
I20251024 19:48:09.651089 2441 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251024 19:48:09.631695 2548 leader_election.cc:290] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:09.673164 1511 raft_consensus.cc:2468] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:09.686501 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fffc42ddda14bd28800917ff2f0ced5" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:09.688447 2548 raft_consensus.cc:493] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:09.688611 2548 raft_consensus.cc:515] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:09.692632 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9287f340fb554904986f6c18b7356701 consensus state has no leader
I20251024 19:48:09.695868 1775 ts_tablet_manager.cc:1929] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251024 19:48:09.696280 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03cdbd1b62b741249ad78562a7edfd22 consensus state has no leader
W20251024 19:48:09.697667 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10891ff5bf0d4b93bcab9faa5f7346f1 consensus state has no leader
I20251024 19:48:09.689306 2548 leader_election.cc:290] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:09.701231 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca0527af718f494a8bbcc66e7b325bed" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:09.701471 1508 raft_consensus.cc:3060] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:09.707604 1338 tablet_service.cc:1505] Processing CreateTablet for tablet d4421b15a60d4256aeaa894bb1e4f99d (DEFAULT_TABLE table=default.Table_bab51dadce54487d96884e3ef240e684 [id=0d7e281e4dd641a69e94a842bffddf0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.714289 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce0ef4443e574846a9f48fef3b893352" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:09.715495 1775 log.cc:1199] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/030e690ce4df4818a6450ae293b0ffbd
I20251024 19:48:09.720072 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4421b15a60d4256aeaa894bb1e4f99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.722510 2441 webserver.cc:492] Webserver started at http://127.0.252.254:42051/ using document root <none> and password file <none>
I20251024 19:48:09.725039 1774 ts_tablet_manager.cc:1929] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:09.725143 1774 log.cc:1199] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/030e690ce4df4818a6450ae293b0ffbd
I20251024 19:48:09.725553 1774 ts_tablet_manager.cc:1950] T 030e690ce4df4818a6450ae293b0ffbd P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:09.726080 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 431) took 4175 ms. Trace:
I20251024 19:48:09.726153 1774 rpcz_store.cc:276] 1024 19:48:05.550812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.550869 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:09.503522 (+3952653us) ts_tablet_manager.cc:1151] Deleting tablet 030e690ce4df4818a6450ae293b0ffbd
1024 19:48:09.503577 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.580501 (+ 76924us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.724935 (+144434us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.726062 (+ 1127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3952195}
I20251024 19:48:09.726550 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 432) took 4161 ms. Trace:
I20251024 19:48:09.726619 1774 rpcz_store.cc:276] 1024 19:48:05.564766 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.565644 (+ 878us) service_pool.cc:225] Handling call
1024 19:48:09.726435 (+4160791us) ts_tablet_manager.cc:1151] Deleting tablet 3e6f8764e9ae472b9ac35a032182aed1
1024 19:48:09.726444 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.726534 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4160483}
I20251024 19:48:09.726918 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 433) took 4103 ms. Trace:
I20251024 19:48:09.726981 1774 rpcz_store.cc:276] 1024 19:48:05.623668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.623730 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:09.726829 (+4103099us) ts_tablet_manager.cc:1151] Deleting tablet 3fd8aa1d9aca429dbfe46f472ebbf53b
1024 19:48:09.726837 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.726904 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4102821}
I20251024 19:48:09.727308 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 434) took 4102 ms. Trace:
I20251024 19:48:09.727370 1774 rpcz_store.cc:276] 1024 19:48:05.625013 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.625076 (+ 63us) service_pool.cc:225] Handling call
1024 19:48:09.727212 (+4102136us) ts_tablet_manager.cc:1151] Deleting tablet 505c30f0b2b749ce918b578903db3757
1024 19:48:09.727219 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.727292 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4078416}
I20251024 19:48:09.727650 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 439) took 4074 ms. Trace:
I20251024 19:48:09.727727 1774 rpcz_store.cc:276] 1024 19:48:05.653230 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.656839 (+ 3609us) service_pool.cc:225] Handling call
1024 19:48:09.727551 (+4070712us) ts_tablet_manager.cc:1151] Deleting tablet 7c880a4af94d4041b8c7cf2066002ee3
1024 19:48:09.727559 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.727635 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4070434}
I20251024 19:48:09.728005 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 440) took 4065 ms. Trace:
I20251024 19:48:09.728077 1774 rpcz_store.cc:276] 1024 19:48:05.662564 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.662618 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:09.727917 (+4065299us) ts_tablet_manager.cc:1151] Deleting tablet 7da1601a87524ef1a8b5069ba6823843
1024 19:48:09.727925 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.727990 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4065052}
I20251024 19:48:09.728338 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 442) took 3999 ms. Trace:
I20251024 19:48:09.728399 1774 rpcz_store.cc:276] 1024 19:48:05.729099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.729154 (+ 55us) service_pool.cc:225] Handling call
1024 19:48:09.728254 (+3999100us) ts_tablet_manager.cc:1151] Deleting tablet 505e4fb0755a4caaa462e5f5154bfee2
1024 19:48:09.728261 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.728323 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3998856}
I20251024 19:48:09.728613 1774 tablet_replica.cc:333] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:09.728799 1774 raft_consensus.cc:2243] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:09.728986 1774 raft_consensus.cc:2272] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:09.730702 2441 fs_manager.cc:362] Metadata directory not provided
I20251024 19:48:09.730911 2441 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 19:48:09.730957 1346 tablet_service.cc:1505] Processing CreateTablet for tablet bcdd47a64e2142cda1f2d20afebd4ce5 (DEFAULT_TABLE table=default.Table_56c54a42391a4397bdbcc0964ed0f9e7 [id=01759ac361e14c15a31441a0ef8db7db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.731990 1342 tablet_service.cc:1505] Processing CreateTablet for tablet b24a3a820c9b496dbf6955b399c6ea8c (DEFAULT_TABLE table=default.Table_434a6ef23bea4f3991418b6ed2d9f635 [id=f24ffafc3bb24a57a21ec7e070a46d03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.732457 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 9198321bcc9e4e47a4c2a5442b6bc850 (DEFAULT_TABLE table=default.Table_af63b6be7a7241feb3512fe1856876bc [id=2c566533af584469a15dd5896e083d34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.736531 1774 ts_tablet_manager.cc:1916] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:09.740031 1340 tablet_service.cc:1505] Processing CreateTablet for tablet af5414e8275845f58717f19fa1b900ed (DEFAULT_TABLE table=default.Table_04c43324857b4aa7b8a4c4f6eb0769f3 [id=3d2816d24b694e609b335150c39d4192]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.740494 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 030e690ce4df4818a6450ae293b0ffbd (table default.Table_4db7a2da5af4481db5b546d201dfd278 [id=2ac38aaa9204452787b18173393bb949]) successfully deleted
W20251024 19:48:09.740814 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3e6f8764e9ae472b9ac35a032182aed1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e6f8764e9ae472b9ac35a032182aed1 already in progress: creating tablet
W20251024 19:48:09.741034 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fd8aa1d9aca429dbfe46f472ebbf53b already in progress: creating tablet
W20251024 19:48:09.741236 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 505c30f0b2b749ce918b578903db3757 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 505c30f0b2b749ce918b578903db3757 already in progress: creating tablet
W20251024 19:48:09.741454 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7c880a4af94d4041b8c7cf2066002ee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c880a4af94d4041b8c7cf2066002ee3 already in progress: creating tablet
W20251024 19:48:09.741696 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7da1601a87524ef1a8b5069ba6823843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7da1601a87524ef1a8b5069ba6823843 already in progress: creating tablet
W20251024 19:48:09.741937 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 505e4fb0755a4caaa462e5f5154bfee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 505e4fb0755a4caaa462e5f5154bfee2 already in progress: creating tablet
I20251024 19:48:09.743688 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b24a3a820c9b496dbf6955b399c6ea8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.744228 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9198321bcc9e4e47a4c2a5442b6bc850. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.745380 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 60f591124dbb444fa7a5b7f408d575df (DEFAULT_TABLE table=default.Table_7065c04cd8e6495180d009455573cde0 [id=06ec4a8cd42a46cab05cbf63a1f6892d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.747363 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 6b9d7c4ecafe40d698187564201582bb (DEFAULT_TABLE table=default.Table_cfeecdf5f1bd4eb099bf97e5713a4d41 [id=34657e83e42f410ca0d4ed4e036d15df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.747718 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 584cbe0c396841f794e665aae3ebbd91 (DEFAULT_TABLE table=default.Table_35c6a6e931bc4596a02d717ead9865ce [id=2cf6b584c9b54f9fad960f515568b21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.747998 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af5414e8275845f58717f19fa1b900ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.750860 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f591124dbb444fa7a5b7f408d575df. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:09.752974 2568 log.cc:927] Time spent T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20251024 19:48:09.762595 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b9d7c4ecafe40d698187564201582bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.762269 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 3a712f52cd114e9bb07c2717e5d728fd (DEFAULT_TABLE table=default.Table_fa23a3c70c3247f9ac84e69f79a41640 [id=b4fdd246321146cdbf968d162389fe50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.767133 1775 ts_tablet_manager.cc:1950] T 030e690ce4df4818a6450ae293b0ffbd P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:09.802029 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 3bacc8601ca1437586bdc9a52e086299 (DEFAULT_TABLE table=default.Table_9fee2e99d42e4cccae5d6e4825802713 [id=63c6383bbaf7493b96eb6e7230695800]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.802349 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4 (DEFAULT_TABLE table=default.Table_d101352faa2b4f4b9b983143a5142043 [id=037cdc76ba24493595393d134223c759]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.803731 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 4846799df5b242ce8f19c034320c8a1d (DEFAULT_TABLE table=default.Table_b1d45a27e5bd42e4a24d1a89f49f9221 [id=fe6f3191d9fb4afbb64b4097912afc89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.772776 1344 tablet_service.cc:1505] Processing CreateTablet for tablet e7255804242d424f8cbf89ef57d4ff9f (DEFAULT_TABLE table=default.Table_7b543dc2852946a78c82d145655196ae [id=191585bc444043ceb08ae1e5f7ee00a0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:09.808386 1483 consensus_meta.cc:287] T c1f4bb2e3a5b406fb5b37403667be5d8 P 10a9536207e343f188e0ef709081d04e: Time spent flushing consensus metadata: real 0.578s user 0.004s sys 0.000s
I20251024 19:48:09.809111 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ad7f12b3760d494db551de925da842df" } modify_external_catalogs: true
I20251024 19:48:09.809388 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ad7f12b3760d494db551de925da842df" } modify_external_catalogs: true
I20251024 19:48:09.810420 1345 tablet_service.cc:1505] Processing CreateTablet for tablet a5442a5604134c4d86cfe6ca730146f0 (DEFAULT_TABLE table=default.Table_733a79b22a954b088ad061fc232e3008 [id=6c5a74f370f14c6588505015c6136507]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.811059 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_86395810be7844a68994a4a5bbf0ab32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:09.811506 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 584cbe0c396841f794e665aae3ebbd91. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:09.811758 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86395810be7844a68994a4a5bbf0ab32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.812114 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcdd47a64e2142cda1f2d20afebd4ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.812289 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a712f52cd114e9bb07c2717e5d728fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.818782 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4c3de3c56f624eee8d86a60154ec6f07" } new_table_name: "default.Table_933a6697624149369ab3ec28462b466a" modify_external_catalogs: true
I20251024 19:48:09.819198 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_65097482f924409b87d7f4dcb5029849" } modify_external_catalogs: true
I20251024 19:48:09.819439 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_65097482f924409b87d7f4dcb5029849" } modify_external_catalogs: true
I20251024 19:48:09.820386 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_773b5ef92fac472a8393530fed39600a" } new_table_name: "default.Table_97ae72e3d285443a8435906795593d52" modify_external_catalogs: true
W20251024 19:48:09.821236 1586 leader_election.cc:343] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:09.821566 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6651e9c63f1d4f19b3d529bff417377b" } modify_external_catalogs: true
W20251024 19:48:09.821642 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 223617us
I20251024 19:48:09.821766 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6651e9c63f1d4f19b3d529bff417377b" } modify_external_catalogs: true
I20251024 19:48:09.822484 2569 raft_consensus.cc:493] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:09.822638 2569 raft_consensus.cc:515] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:09.822813 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 57e1e21beee44847b1ebeb746f02c372 (DEFAULT_TABLE table=default.Table_a6498b82a207414dafe80255fff08e8c [id=73c28b9621534cb896be0badbe3911be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.823101 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b9cb3ccac55f4b0a8897f5cd1412440d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:09.823251 2569 leader_election.cc:290] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:09.823627 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9cb3ccac55f4b0a8897f5cd1412440d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.823644 2569 consensus_queue.cc:1048] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:09.824012 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.825012 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e97c30902b564ba2a411ea987cb0ef70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:09.825357 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ff7a43292b9045f9bbfa71883bcd7e61" } modify_external_catalogs: true
W20251024 19:48:09.825537 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e97c30902b564ba2a411ea987cb0ef70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.825559 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ff7a43292b9045f9bbfa71883bcd7e61" } modify_external_catalogs: true
I20251024 19:48:09.828712 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2d39eb2f825b417a9361c6f4876e0cd7
I20251024 19:48:09.831483 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_88d611a9d596483bb568a4fbdcf71f17" } modify_external_catalogs: true
I20251024 19:48:09.831661 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_88d611a9d596483bb568a4fbdcf71f17" } modify_external_catalogs: true
I20251024 19:48:09.832232 2501 raft_consensus.cc:493] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:09.832394 2501 raft_consensus.cc:515] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:09.833005 2501 leader_election.cc:290] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:09.833281 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8359a8d5bf544cca9b2407a650f3c9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.841215 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet 59956dad06064024ab171631ef6c1dca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:09.841465 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet c0f231f739b64f9fb6a726589a213146
W20251024 19:48:09.841951 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 635a132735704cfda69bf7c8deaea81d consensus state has no leader
W20251024 19:48:09.842381 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e03c488d0f264684b0897a81a04d7466 consensus state has no leader
I20251024 19:48:09.847486 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet fd52295b14b14e70bbdcb45f1bd849e4
I20251024 19:48:09.848114 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7be9290daabb404f955f05922d897fbf
I20251024 19:48:09.849973 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7e0dc7d4fed5448cb616af29e405e766
I20251024 19:48:09.851975 1768 tablet_bootstrap.cc:492] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:09.852121 1768 ts_tablet_manager.cc:1403] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.445s user 0.017s sys 0.027s
I20251024 19:48:09.852842 1768 raft_consensus.cc:359] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:09.853008 1768 raft_consensus.cc:385] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:09.853057 1768 raft_consensus.cc:740] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:09.853235 1768 consensus_queue.cc:260] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:09.853726 1768 ts_tablet_manager.cc:1434] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:09.853771 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet d2c92975cf66441f9ac90c19772d3db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:09.859828 1776 ts_tablet_manager.cc:1929] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:09.859951 1776 log.cc:1199] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/fca73ab61f0c47df9186459caf2b9a32
I20251024 19:48:09.860288 1776 ts_tablet_manager.cc:1950] T fca73ab61f0c47df9186459caf2b9a32 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:09.861068 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 565) took 2121 ms. Trace:
I20251024 19:48:09.861143 1776 rpcz_store.cc:276] 1024 19:48:07.740876 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.740944 (+ 68us) service_pool.cc:225] Handling call
1024 19:48:09.015155 (+1274211us) ts_tablet_manager.cc:1151] Deleting tablet fca73ab61f0c47df9186459caf2b9a32
1024 19:48:09.015169 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.260577 (+245408us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.859735 (+599158us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.861051 (+ 1316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205041}
I20251024 19:48:09.861521 1776 tablet_replica.cc:333] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:09.861735 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce0ef4443e574846a9f48fef3b893352" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:09.861872 1617 tablet_service.cc:1505] Processing CreateTablet for tablet bcdd47a64e2142cda1f2d20afebd4ce5 (DEFAULT_TABLE table=default.Table_56c54a42391a4397bdbcc0964ed0f9e7 [id=01759ac361e14c15a31441a0ef8db7db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.862282 1626 tablet_service.cc:1505] Processing CreateTablet for tablet d4421b15a60d4256aeaa894bb1e4f99d (DEFAULT_TABLE table=default.Table_bab51dadce54487d96884e3ef240e684 [id=0d7e281e4dd641a69e94a842bffddf0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.862396 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 60f591124dbb444fa7a5b7f408d575df (DEFAULT_TABLE table=default.Table_7065c04cd8e6495180d009455573cde0 [id=06ec4a8cd42a46cab05cbf63a1f6892d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.862780 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet 2087e295ad4b4daa8eeebd49b43e08c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:09.863082 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 9198321bcc9e4e47a4c2a5442b6bc850 (DEFAULT_TABLE table=default.Table_af63b6be7a7241feb3512fe1856876bc [id=2c566533af584469a15dd5896e083d34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.862789 1619 tablet_service.cc:1505] Processing CreateTablet for tablet b24a3a820c9b496dbf6955b399c6ea8c (DEFAULT_TABLE table=default.Table_434a6ef23bea4f3991418b6ed2d9f635 [id=f24ffafc3bb24a57a21ec7e070a46d03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.863308 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f89a3bbeb9b4d57a49078729110d951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:09.866549 1630 tablet_service.cc:1505] Processing CreateTablet for tablet a5442a5604134c4d86cfe6ca730146f0 (DEFAULT_TABLE table=default.Table_733a79b22a954b088ad061fc232e3008 [id=6c5a74f370f14c6588505015c6136507]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.866886 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 64499c8734d34c66b0b52a2cd3ef0673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:09.867192 1622 tablet_service.cc:1505] Processing CreateTablet for tablet af5414e8275845f58717f19fa1b900ed (DEFAULT_TABLE table=default.Table_04c43324857b4aa7b8a4c4f6eb0769f3 [id=3d2816d24b694e609b335150c39d4192]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.867754 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d8da45c6364f1e9ddc79307904587c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.867993 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b24a3a820c9b496dbf6955b399c6ea8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.872460 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4421b15a60d4256aeaa894bb1e4f99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.873550 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f591124dbb444fa7a5b7f408d575df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.876910 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcdd47a64e2142cda1f2d20afebd4ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.884388 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9198321bcc9e4e47a4c2a5442b6bc850. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.886942 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5442a5604134c4d86cfe6ca730146f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.894428 1776 raft_consensus.cc:2243] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:09.894843 1776 pending_rounds.cc:70] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:09.894955 1776 pending_rounds.cc:77] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429340838838272 op_type: NO_OP noop_request { }
I20251024 19:48:09.895152 1776 raft_consensus.cc:2889] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:09.895399 1776 raft_consensus.cc:2272] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:09.896222 1776 ts_tablet_manager.cc:1916] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:09.902426 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_56bed2d1fdc64b03a33e778185a2f371"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:09.902710 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fe00154d4ddf4d90a36ab30b672cc5cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.903241 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe00154d4ddf4d90a36ab30b672cc5cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:09.903297 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56bed2d1fdc64b03a33e778185a2f371 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.904543 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_645f2377cbe34c8dbacd9c416c00cfa1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:09.904722 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3c5c40909920463ab35ba50c7a03bd36" } new_table_name: "default.Table_2608adc41cfa4b96bc13dee99a9fca7b" modify_external_catalogs: true
W20251024 19:48:09.905063 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_645f2377cbe34c8dbacd9c416c00cfa1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.905707 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_928046849b864c76b36b8a2526f9bc5e" } new_table_name: "default.Table_575ba6a1b4c64e77aadef06e3084a52a" modify_external_catalogs: true
I20251024 19:48:09.907424 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d343821349d34633bfa9da8bc9dad031"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.907964 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d343821349d34633bfa9da8bc9dad031 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.908528 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fdc631c49a3d4e6b89b84a4af697b305"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:09.909024 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a7d96771ce8440ebb201c4095b80c7c8" } new_table_name: "default.Table_b733659f0fe548d78e86ae0925c680a4" modify_external_catalogs: true
W20251024 19:48:09.909291 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdc631c49a3d4e6b89b84a4af697b305 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.910050 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7e1ad08d4bd3477494cf5688f7842dd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:09.910739 2441 fs_manager.cc:714] Time spent opening directory manager: real 0.157s user 0.027s sys 0.017s
W20251024 19:48:09.912468 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e1ad08d4bd3477494cf5688f7842dd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.909373 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_49ca74f7a5d24c139a894b1bb06f7f5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:09.916507 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 672b571081ea4cd78028b6a25625103c consensus state has no leader
I20251024 19:48:09.922329 1774 ts_tablet_manager.cc:1929] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:09.922463 1774 log.cc:1199] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/aebf76d4a66947ed9756643740418798
I20251024 19:48:09.922859 1774 ts_tablet_manager.cc:1950] T aebf76d4a66947ed9756643740418798 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:48:09.926872 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49ca74f7a5d24c139a894b1bb06f7f5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:09.931761 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4846799df5b242ce8f19c034320c8a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.932523 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bacc8601ca1437586bdc9a52e086299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.934660 2586 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 19:48:09.938441 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af5414e8275845f58717f19fa1b900ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.939265 2441 fs_manager.cc:730] Time spent opening block manager: real 0.026s user 0.011s sys 0.009s
I20251024 19:48:09.940078 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4 (DEFAULT_TABLE table=default.Table_d101352faa2b4f4b9b983143a5142043 [id=037cdc76ba24493595393d134223c759]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.940495 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet fca73ab61f0c47df9186459caf2b9a32 (table default.Table_420ebc050cf440e9a59a18e26f7b83af [id=be1cc9d047554ea6a9afa531fb08a81d]) successfully deleted
I20251024 19:48:09.939494 2441 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
uuid: "ef8906492b1a4368936fac95c11e1f73"
format_stamp: "Formatted at 2025-10-24 19:47:51 on dist-test-slave-53sn"
I20251024 19:48:09.940747 2501 raft_consensus.cc:493] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:09.940879 2501 raft_consensus.cc:515] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:09.940716 2441 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 19:48:09.940987 1305 leader_election.cc:304] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:09.942291 1627 tablet_service.cc:1505] Processing CreateTablet for tablet 584cbe0c396841f794e665aae3ebbd91 (DEFAULT_TABLE table=default.Table_35c6a6e931bc4596a02d717ead9865ce [id=2cf6b584c9b54f9fad960f515568b21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.943606 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 6b9d7c4ecafe40d698187564201582bb (DEFAULT_TABLE table=default.Table_cfeecdf5f1bd4eb099bf97e5713a4d41 [id=34657e83e42f410ca0d4ed4e036d15df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.944190 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b9d7c4ecafe40d698187564201582bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.945286 1621 tablet_service.cc:1505] Processing CreateTablet for tablet e7255804242d424f8cbf89ef57d4ff9f (DEFAULT_TABLE table=default.Table_7b543dc2852946a78c82d145655196ae [id=191585bc444043ceb08ae1e5f7ee00a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.945844 2501 leader_election.cc:290] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:09.946333 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.946609 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7255804242d424f8cbf89ef57d4ff9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.947602 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 584cbe0c396841f794e665aae3ebbd91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.947867 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 57e1e21beee44847b1ebeb746f02c372 (DEFAULT_TABLE table=default.Table_a6498b82a207414dafe80255fff08e8c [id=73c28b9621534cb896be0badbe3911be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.948168 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e1e21beee44847b1ebeb746f02c372. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.948887 1626 tablet_service.cc:1505] Processing CreateTablet for tablet 3bacc8601ca1437586bdc9a52e086299 (DEFAULT_TABLE table=default.Table_9fee2e99d42e4cccae5d6e4825802713 [id=63c6383bbaf7493b96eb6e7230695800]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.949206 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bacc8601ca1437586bdc9a52e086299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.949287 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 3a712f52cd114e9bb07c2717e5d728fd (DEFAULT_TABLE table=default.Table_fa23a3c70c3247f9ac84e69f79a41640 [id=b4fdd246321146cdbf968d162389fe50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.949559 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a712f52cd114e9bb07c2717e5d728fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.954327 2548 raft_consensus.cc:2749] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:09.955229 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7255804242d424f8cbf89ef57d4ff9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.955668 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e1e21beee44847b1ebeb746f02c372. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.957142 1616 tablet_service.cc:1505] Processing CreateTablet for tablet e602f14c45104517b7618b163519a499 (DEFAULT_TABLE table=default.Table_d10102cfa0fa4eb9b806bacdd5066807 [id=de0ca8d2f97b4354801830c6b614db9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.967604 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e602f14c45104517b7618b163519a499. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.971982 1766 tablet_bootstrap.cc:492] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:09.972164 1766 ts_tablet_manager.cc:1403] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.915s user 0.017s sys 0.013s
I20251024 19:48:09.972110 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 4846799df5b242ce8f19c034320c8a1d (DEFAULT_TABLE table=default.Table_b1d45a27e5bd42e4a24d1a89f49f9221 [id=fe6f3191d9fb4afbb64b4097912afc89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.972855 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 7cab9c17496947efbbe3cd9e5d8a49a8 (DEFAULT_TABLE table=default.Table_fc2a1aeddc444f578693d922323862f5 [id=8a834f513c0045ca874a0d76e08d494c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.973194 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cab9c17496947efbbe3cd9e5d8a49a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.976219 1624 tablet_service.cc:1505] Processing CreateTablet for tablet bf241a9566c34b7b911b69aaf4de06a2 (DEFAULT_TABLE table=default.Table_3abf70710bce4728bfecd4a89d31dea6 [id=19704cecc9eb4063a579e16c491641df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:09.976595 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf241a9566c34b7b911b69aaf4de06a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.865597 2561 consensus_queue.cc:1048] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.113s
I20251024 19:48:09.981103 1766 raft_consensus.cc:359] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:09.981350 1766 raft_consensus.cc:385] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:09.981415 1766 raft_consensus.cc:740] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:09.981679 1766 consensus_queue.cc:260] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:09.982304 1766 ts_tablet_manager.cc:1434] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251024 19:48:09.987445 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4846799df5b242ce8f19c034320c8a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:09.995770 1766 tablet_bootstrap.cc:492] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:09.969807 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5442a5604134c4d86cfe6ca730146f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:09.970219 1305 leader_election.cc:343] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 9fffc42ddda14bd28800917ff2f0ced5
I20251024 19:48:10.010711 1305 leader_election.cc:304] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:09.891039 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 485) took 3897 ms. Trace:
I20251024 19:48:10.017863 2583 raft_consensus.cc:2749] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:10.019894 1305 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 49684.4us
I20251024 19:48:10.021890 2441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 19:48:10.024940 1350 tablet_service.cc:1505] Processing CreateTablet for tablet e602f14c45104517b7618b163519a499 (DEFAULT_TABLE table=default.Table_d10102cfa0fa4eb9b806bacdd5066807 [id=de0ca8d2f97b4354801830c6b614db9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.025349 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e602f14c45104517b7618b163519a499. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.025864 2441 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 19:48:10.026350 2441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 19:48:10.031374 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d39eb2f825b417a9361c6f4876e0cd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:10.032795 1623 tablet_service.cc:1505] Processing CreateTablet for tablet e51e597b8fd044e393a3af07da25586c (DEFAULT_TABLE table=default.Table_1dbcae61809349c59e959bff6c5cdf24 [id=64c3c3285f5a4c36bc31c5577e3695cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.061548 1329 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:54172 (request call id 123) took 1046 ms. Trace:
W20251024 19:48:10.062935 1334 consensus_meta.cc:287] T 8359a8d5bf544cca9b2407a650f3c9b1 P 43bec2f03fc849099e254f6335e9d532: Time spent flushing consensus metadata: real 0.546s user 0.003s sys 0.004s
I20251024 19:48:10.016626 1775 rpcz_store.cc:276] 1024 19:48:05.993258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.993304 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:08.897171 (+2903867us) ts_tablet_manager.cc:1151] Deleting tablet 030e690ce4df4818a6450ae293b0ffbd
1024 19:48:08.897182 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.162081 (+264899us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.695758 (+533677us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.891010 (+195252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2903574}
I20251024 19:48:10.061707 1329 rpcz_store.cc:276] 1024 19:48:09.014630 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.014683 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:09.568752 (+554069us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x6160001a1d54. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:10.061517 (+492765us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1238193536}
I20251024 19:48:10.074082 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 7cab9c17496947efbbe3cd9e5d8a49a8 (DEFAULT_TABLE table=default.Table_fc2a1aeddc444f578693d922323862f5 [id=8a834f513c0045ca874a0d76e08d494c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.039673 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet c0f231f739b64f9fb6a726589a213146 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:10.042876 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet fd52295b14b14e70bbdcb45f1bd849e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:10.055306 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 7be9290daabb404f955f05922d897fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:10.056757 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e0dc7d4fed5448cb616af29e405e766 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:10.091315 1348 tablet_service.cc:1505] Processing CreateTablet for tablet bf241a9566c34b7b911b69aaf4de06a2 (DEFAULT_TABLE table=default.Table_3abf70710bce4728bfecd4a89d31dea6 [id=19704cecc9eb4063a579e16c491641df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.098964 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 482) took 4105 ms. Trace:
I20251024 19:48:10.099089 1775 rpcz_store.cc:276] 1024 19:48:05.992977 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.993514 (+ 537us) service_pool.cc:225] Handling call
1024 19:48:10.066877 (+4073363us) ts_tablet_manager.cc:1151] Deleting tablet 2496f103436047dea5dabb577651249f
1024 19:48:10.066888 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.098810 (+ 31922us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4073154}
I20251024 19:48:10.061535 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e51e597b8fd044e393a3af07da25586c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.100853 2561 raft_consensus.cc:493] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:10.061131 1628 tablet_service.cc:1505] Processing CreateTablet for tablet c0f0c6c4a3f04b079a2a2c0b1b2050b7 (DEFAULT_TABLE table=default.Table_86395810be7844a68994a4a5bbf0ab32 [id=b5a5fe39570b4001b750c392d2823d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.100978 2561 raft_consensus.cc:515] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.101241 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f0c6c4a3f04b079a2a2c0b1b2050b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.101687 2561 leader_election.cc:290] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:10.107986 1767 tablet_bootstrap.cc:654] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.108210 1334 tablet_service.cc:1505] Processing CreateTablet for tablet e51e597b8fd044e393a3af07da25586c (DEFAULT_TABLE table=default.Table_1dbcae61809349c59e959bff6c5cdf24 [id=64c3c3285f5a4c36bc31c5577e3695cf]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:10.111430 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 289128us
W20251024 19:48:10.111788 1447 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.751s user 0.172s sys 0.323s
W20251024 19:48:10.111925 1447 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.751s user 0.172s sys 0.323s
I20251024 19:48:10.112080 1586 leader_election.cc:304] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:10.112571 1447 leader_election.cc:343] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.113150 1775 tablet_replica.cc:333] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:10.114054 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bf08ccf7054fedb9efcbb00354c71a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:10.114450 2584 raft_consensus.cc:2804] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:10.114526 2584 raft_consensus.cc:493] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:10.114590 2584 raft_consensus.cc:3060] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:10.115301 2569 consensus_queue.cc:1048] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:10.116559 2501 consensus_queue.cc:1048] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:10.118242 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 443) took 4381 ms. Trace:
I20251024 19:48:10.118551 1768 tablet_bootstrap.cc:492] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:10.119654 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c55267a23ded43cab7442fa75b611e12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:10.120285 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c55267a23ded43cab7442fa75b611e12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:10.123132 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4798dd4a61224784b9963dd87de7a19a" } modify_external_catalogs: true
I20251024 19:48:10.123314 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4798dd4a61224784b9963dd87de7a19a" } modify_external_catalogs: true
I20251024 19:48:10.123939 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_370e897b37284a38948769e22c2d22f3" } modify_external_catalogs: true
I20251024 19:48:10.124106 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_370e897b37284a38948769e22c2d22f3" } modify_external_catalogs: true
W20251024 19:48:10.126142 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 consensus state has no leader
I20251024 19:48:10.128603 1447 leader_election.cc:304] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:10.129096 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6cc41f0070cf4f22b3ce19c688e77c50" } new_table_name: "default.Table_aea1b9672b8c4d1698e782c6824de441" modify_external_catalogs: true
I20251024 19:48:10.118327 1774 rpcz_store.cc:276] 1024 19:48:05.737079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.737135 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:09.728576 (+3991441us) ts_tablet_manager.cc:1151] Deleting tablet aebf76d4a66947ed9756643740418798
1024 19:48:09.728584 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:09.764419 (+ 35835us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:09.922220 (+157801us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.118221 (+196001us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3991232}
I20251024 19:48:10.134280 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet aebf76d4a66947ed9756643740418798 (table default.Table_d09f149249c44c50adcb71321c133f2d [id=665389c7bde54b7786edabcb91a9b33b]) successfully deleted
I20251024 19:48:10.135250 1508 raft_consensus.cc:2468] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:10.136293 2584 raft_consensus.cc:515] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.137020 2584 leader_election.cc:290] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:10.138043 1444 leader_election.cc:343] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:10.138274 1586 leader_election.cc:343] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.139853 2501 consensus_queue.cc:1048] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:10.140712 1586 leader_election.cc:343] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:10.142408 1586 consensus_peers.cc:597] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:10.143069 2501 consensus_queue.cc:1048] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:10.148407 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 453) took 3779 ms. Trace:
I20251024 19:48:10.150637 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_35c6a6e931bc4596a02d717ead9865ce" } new_table_name: "default.Table_e9accb7540aa4f2ebe096fd57e7e5105" modify_external_catalogs: true
I20251024 19:48:10.154270 1776 ts_tablet_manager.cc:1929] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:10.154392 1776 log.cc:1199] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/5b96fc2309274048bed7a35e4bf4bae2
I20251024 19:48:10.154781 1776 ts_tablet_manager.cc:1950] T 5b96fc2309274048bed7a35e4bf4bae2 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:10.155630 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 577) took 2206 ms. Trace:
I20251024 19:48:10.155704 1776 rpcz_store.cc:276] 1024 19:48:07.948679 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.951282 (+ 2603us) service_pool.cc:225] Handling call
1024 19:48:09.861475 (+1910193us) ts_tablet_manager.cc:1151] Deleting tablet 5b96fc2309274048bed7a35e4bf4bae2
1024 19:48:09.861485 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.069054 (+207569us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.154151 (+ 85097us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.155606 (+ 1455us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1898092}
I20251024 19:48:10.156330 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 5b96fc2309274048bed7a35e4bf4bae2 (table default.Table_dc366a1f70d84d4f8276c1872b9fa0f7 [id=0f190205e9b347f6906d6f86b0624681]) successfully deleted
I20251024 19:48:10.156998 1766 tablet_bootstrap.cc:654] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.159966 2561 raft_consensus.cc:2749] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:10.160233 1447 consensus_peers.cc:597] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:10.160439 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_34c8fbbe53954b97b7ead671ade3783b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:10.161361 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34c8fbbe53954b97b7ead671ade3783b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:10.161538 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7503b22d9e12481b9af96385b9d76e50" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:10.161777 1508 raft_consensus.cc:2468] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:10.161978 1775 raft_consensus.cc:2243] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:10.162163 1775 raft_consensus.cc:2272] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.162218 1586 leader_election.cc:304] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:10.162801 1775 ts_tablet_manager.cc:1916] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.163491 2501 raft_consensus.cc:2804] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:10.163622 2501 raft_consensus.cc:697] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:10.165146 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2037fcca87b94b5fb31639b4d07307b0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:10.166914 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3259823b622461fa823841925b3d203" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:10.167151 1508 raft_consensus.cc:2468] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:10.172375 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_72be3d9212c64e6dbe03acba1f12df24" } modify_external_catalogs: true
I20251024 19:48:10.172688 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_72be3d9212c64e6dbe03acba1f12df24" } modify_external_catalogs: true
I20251024 19:48:10.173502 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 030e690ce4df4818a6450ae293b0ffbd (table default.Table_4db7a2da5af4481db5b546d201dfd278 [id=2ac38aaa9204452787b18173393bb949]) successfully deleted
W20251024 19:48:10.174194 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2496f103436047dea5dabb577651249f because the tablet was not found. No further retry: Not found: Tablet not found: 2496f103436047dea5dabb577651249f
I20251024 19:48:10.176497 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bf08ccf7054fedb9efcbb00354c71a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:10.177064 1372 raft_consensus.cc:3060] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:10.178336 1776 tablet_replica.cc:333] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:10.183732 1776 raft_consensus.cc:2243] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:10.163885 2501 consensus_queue.cc:237] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:10.189617 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6471c185917e4751962898ec47aa725b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:10.190307 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6471c185917e4751962898ec47aa725b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:10.193645 1616 tablet_service.cc:1505] Processing CreateTablet for tablet de2ebb79d4fe4a1bbac372bb26f370ed (DEFAULT_TABLE table=default.Table_b9cb3ccac55f4b0a8897f5cd1412440d [id=c329ff0d97454526835cdda8938deeb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.193464 1586 leader_election.cc:304] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:10.195307 2584 raft_consensus.cc:2804] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:10.195401 2584 raft_consensus.cc:493] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:10.195478 2584 raft_consensus.cc:3060] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:10.195952 1509 raft_consensus.cc:2468] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:10.197752 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ace1982961c34807a364bd4f35907799" } modify_external_catalogs: true
I20251024 19:48:10.197988 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ace1982961c34807a364bd4f35907799" } modify_external_catalogs: true
I20251024 19:48:10.200229 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2037fcca87b94b5fb31639b4d07307b0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:10.200712 1776 pending_rounds.cc:70] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:10.200718 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3259823b622461fa823841925b3d203" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:10.200793 1776 pending_rounds.cc:77] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429342251991040 op_type: NO_OP noop_request { }
I20251024 19:48:10.200954 1776 raft_consensus.cc:2889] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:10.201090 1776 raft_consensus.cc:2272] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.208668 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d39eb2f825b417a9361c6f4876e0cd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:10.208580 1774 rpcz_store.cc:276] 1024 19:48:06.369092 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.369636 (+ 544us) service_pool.cc:225] Handling call
1024 19:48:10.133139 (+3763503us) ts_tablet_manager.cc:1151] Deleting tablet 780442b310e6409498e6bbe4256280a4
1024 19:48:10.148235 (+ 15096us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.148386 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3763207}
I20251024 19:48:10.208915 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet c0f231f739b64f9fb6a726589a213146 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:10.212358 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet fd52295b14b14e70bbdcb45f1bd849e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:10.215857 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cab9c17496947efbbe3cd9e5d8a49a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.216142 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf241a9566c34b7b911b69aaf4de06a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.217087 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7503b22d9e12481b9af96385b9d76e50" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:10.218997 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e51e597b8fd044e393a3af07da25586c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.225211 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet fca73ab61f0c47df9186459caf2b9a32 with delete_type TABLET_DATA_DELETED (replaced by tablet f83b9290f00145a780318a43249eec12) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:10.225692 1349 tablet_service.cc:1505] Processing CreateTablet for tablet de2ebb79d4fe4a1bbac372bb26f370ed (DEFAULT_TABLE table=default.Table_b9cb3ccac55f4b0a8897f5cd1412440d [id=c329ff0d97454526835cdda8938deeb8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:10.226251 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 780442b310e6409498e6bbe4256280a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 780442b310e6409498e6bbe4256280a4 already in progress: creating tablet
I20251024 19:48:10.232175 1768 tablet_bootstrap.cc:654] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.232921 1351 tablet_service.cc:1552] Processing DeleteTablet for tablet 7be9290daabb404f955f05922d897fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:10.232966 1484 tablet_service.cc:1505] Processing CreateTablet for tablet 8359a8d5bf544cca9b2407a650f3c9b1 (DEFAULT_TABLE table=default.Table_6eab640e6ecf469aa77cd34cbfa7b361 [id=e5179c71774c47938da28d18b02cc812]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.233443 1485 tablet_service.cc:1505] Processing CreateTablet for tablet f4e812c43ade4831af50825474976c8e (DEFAULT_TABLE table=default.Table_bd37f8f763d147b2b7656b63f72fd749 [id=91696606f250441eaeade9c5cad1cc5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.233554 1484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8359a8d5bf544cca9b2407a650f3c9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.233851 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4e812c43ade4831af50825474976c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.234015 1478 tablet_service.cc:1505] Processing CreateTablet for tablet 64d8da45c6364f1e9ddc79307904587c (DEFAULT_TABLE table=default.Table_379c43ee27bd4c74ae00adf4df216fa5 [id=9d26ddec659e4b9ba0530c10dec44397]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.234352 1478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d8da45c6364f1e9ddc79307904587c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.235006 1477 tablet_service.cc:1505] Processing CreateTablet for tablet fce7de9523fb4878a58aa8aef0c6b182 (DEFAULT_TABLE table=default.Table_628b2afa9401426980be0eae5dffd917 [id=5b96ec1514ae41e3a0103e4b381aab61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.235562 1487 tablet_service.cc:1505] Processing CreateTablet for tablet ce5b2cfcc8294b3ea8b89334786006f8 (DEFAULT_TABLE table=default.Table_f57b3f3656d347a3aee8146fd991ca1e [id=5cf6d30527544e698e0fd68ad20c66eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.235950 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce5b2cfcc8294b3ea8b89334786006f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:10.236653 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4a66b7c89794807826bea29229a6d3a consensus state has no leader
I20251024 19:48:10.237304 1480 tablet_service.cc:1505] Processing CreateTablet for tablet 57fed826b4b64edba48c5fffaef1e031 (DEFAULT_TABLE table=default.Table_6501382adb09425e8fa508d81fe78a24 [id=c5a5612d649d415dac70547f7a1e7f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.237728 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57fed826b4b64edba48c5fffaef1e031. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.241865 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 455) took 3848 ms. Trace:
I20251024 19:48:10.243108 1489 tablet_service.cc:1505] Processing CreateTablet for tablet cbd321c059e34834a6257fab5e6d14bb (DEFAULT_TABLE table=default.Table_345aac1447b347f99accafba7777859a [id=0b6b51df83d54f6986b2b53791d2083b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.243252 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 9377725db312439e8a42dbd46da84a9d (DEFAULT_TABLE table=default.Table_f186552a977c4a1e8002ece18abd9d07 [id=2d70ca56de444b8bbc638451232f591b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.243479 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd321c059e34834a6257fab5e6d14bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.243672 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9377725db312439e8a42dbd46da84a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.244760 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 64499c8734d34c66b0b52a2cd3ef0673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.245219 1475 tablet_service.cc:1505] Processing CreateTablet for tablet a7ffb227901f49b99fb57c5c6e138c8c (DEFAULT_TABLE table=default.Table_9ebd164c5e924a7eaff697af91c4c1a5 [id=6a388834945a480dabf5191d604d509f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.245657 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7ffb227901f49b99fb57c5c6e138c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.248016 1336 tablet_service.cc:1505] Processing CreateTablet for tablet c0f0c6c4a3f04b079a2a2c0b1b2050b7 (DEFAULT_TABLE table=default.Table_86395810be7844a68994a4a5bbf0ab32 [id=b5a5fe39570b4001b750c392d2823d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.248237 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_70a5bb93919a463f858c20dfe378fe58" } modify_external_catalogs: true
I20251024 19:48:10.248522 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f0c6c4a3f04b079a2a2c0b1b2050b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.248481 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_70a5bb93919a463f858c20dfe378fe58" } modify_external_catalogs: true
W20251024 19:48:10.249130 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93d84776848049c8a146998219986a84 consensus state has no leader
I20251024 19:48:10.249658 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fc16485e80804cc990dbd6f6aca14502" } new_table_name: "default.Table_98eaa0ba734844c68ebade89470db3d3" modify_external_catalogs: true
I20251024 19:48:10.250380 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7fa0ffaa80294f7eb87d8b82770bb44c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:10.250927 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fa0ffaa80294f7eb87d8b82770bb44c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:10.251891 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fce7de9523fb4878a58aa8aef0c6b182. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.253273 1774 rpcz_store.cc:276] 1024 19:48:06.369390 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.370038 (+ 648us) service_pool.cc:225] Handling call
1024 19:48:10.208997 (+3838959us) ts_tablet_manager.cc:1151] Deleting tablet 3324755a5a184a609fae5f7a25d58e4c
1024 19:48:10.209013 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.217974 (+ 8961us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3836785}
I20251024 19:48:10.253907 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 454) took 3884 ms. Trace:
I20251024 19:48:10.254107 1774 rpcz_store.cc:276] 1024 19:48:06.369216 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.369974 (+ 758us) service_pool.cc:225] Handling call
1024 19:48:10.253713 (+3883739us) ts_tablet_manager.cc:1151] Deleting tablet 65424b00ab9c454f9584048caba83e6a
1024 19:48:10.253725 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.253855 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3877417}
I20251024 19:48:10.254513 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 456) took 3884 ms. Trace:
I20251024 19:48:10.253273 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 839cef50795840fab564b94d88565f8c (DEFAULT_TABLE table=default.Table_e97c30902b564ba2a411ea987cb0ef70 [id=e95322d516ba4098a327e991c8834bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.258525 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de2ebb79d4fe4a1bbac372bb26f370ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.259701 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 839cef50795840fab564b94d88565f8c (DEFAULT_TABLE table=default.Table_e97c30902b564ba2a411ea987cb0ef70 [id=e95322d516ba4098a327e991c8834bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.260377 1486 tablet_service.cc:1505] Processing CreateTablet for tablet f9d09a94d1ea4af19b3da5155b3c4aad (DEFAULT_TABLE table=default.Table_7837f9d39d7e4658a8945b4ddd2cf236 [id=1370e40179c5440185786cfae2bbbd7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.261060 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 839cef50795840fab564b94d88565f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.263518 2584 raft_consensus.cc:515] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.264156 2584 leader_election.cc:290] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:10.264556 1491 tablet_service.cc:1505] Processing CreateTablet for tablet d2c92975cf66441f9ac90c19772d3db3 (DEFAULT_TABLE table=default.Table_1606f259eb4f47678e56892bf267bf77 [id=342122e08a644b7c8758b89d588e3d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.265929 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9d09a94d1ea4af19b3da5155b3c4aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.267206 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c92975cf66441f9ac90c19772d3db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.269758 1776 ts_tablet_manager.cc:1916] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:10.272051 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 584cbe0c396841f794e665aae3ebbd91 consensus state has no leader
I20251024 19:48:10.272581 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_075cb16ace064e7a8840d0d6d2ede7ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:10.273217 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_075cb16ace064e7a8840d0d6d2ede7ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:10.274503 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3324755a5a184a609fae5f7a25d58e4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3324755a5a184a609fae5f7a25d58e4c already in progress: creating tablet
I20251024 19:48:10.275039 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f89a3bbeb9b4d57a49078729110d951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.275568 1482 tablet_service.cc:1505] Processing CreateTablet for tablet d4421b15a60d4256aeaa894bb1e4f99d (DEFAULT_TABLE table=default.Table_bab51dadce54487d96884e3ef240e684 [id=0d7e281e4dd641a69e94a842bffddf0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.252940 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73ffe342f541447d8519ee24bc98a0f3" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:10.283798 1776 ts_tablet_manager.cc:1929] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:10.284046 1776 log.cc:1199] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/3754540ae21e4c65b5af3d0aa3f9189f
I20251024 19:48:10.289654 1774 rpcz_store.cc:276] 1024 19:48:06.369559 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.370061 (+ 502us) service_pool.cc:225] Handling call
1024 19:48:10.254349 (+3884288us) ts_tablet_manager.cc:1151] Deleting tablet 7eb62791100f477784ab315923b6ceb6
1024 19:48:10.254358 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.254445 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3877882}
I20251024 19:48:10.290216 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 460) took 3838 ms. Trace:
I20251024 19:48:10.290336 1774 rpcz_store.cc:276] 1024 19:48:06.451783 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.451937 (+ 154us) service_pool.cc:225] Handling call
1024 19:48:10.290017 (+3838080us) ts_tablet_manager.cc:1151] Deleting tablet 94b35252e63f4a8ba305cc029ddb5ae8
1024 19:48:10.290033 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.290158 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3837877}
I20251024 19:48:10.290761 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 461) took 3838 ms. Trace:
I20251024 19:48:10.290869 1774 rpcz_store.cc:276] 1024 19:48:06.451885 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.452181 (+ 296us) service_pool.cc:225] Handling call
1024 19:48:10.290619 (+3838438us) ts_tablet_manager.cc:1151] Deleting tablet 83b91a5d764449d5b82ef6d1ee0360f5
1024 19:48:10.290629 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.290715 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3838249}
I20251024 19:48:10.291236 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 476) took 3481 ms. Trace:
I20251024 19:48:10.291340 1774 rpcz_store.cc:276] 1024 19:48:06.810257 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.810319 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:10.291101 (+3480782us) ts_tablet_manager.cc:1151] Deleting tablet f88390f2534e4799a6eaa56aa4245e97
1024 19:48:10.291112 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.291188 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3480545}
I20251024 19:48:10.291724 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 483) took 3021 ms. Trace:
I20251024 19:48:10.291824 1774 rpcz_store.cc:276] 1024 19:48:07.269980 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.270102 (+ 122us) service_pool.cc:225] Handling call
1024 19:48:10.291591 (+3021489us) ts_tablet_manager.cc:1151] Deleting tablet 7deb97be82964ea0b2fe9caa867a6460
1024 19:48:10.291601 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.291681 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2997579}
I20251024 19:48:10.292125 1774 tablet_replica.cc:333] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:10.292378 1774 raft_consensus.cc:2243] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:10.292817 1774 raft_consensus.cc:2272] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.293711 1347 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e0dc7d4fed5448cb616af29e405e766 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:10.274868 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73ffe342f541447d8519ee24bc98a0f3" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:10.294703 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 9198321bcc9e4e47a4c2a5442b6bc850 (DEFAULT_TABLE table=default.Table_af63b6be7a7241feb3512fe1856876bc [id=2c566533af584469a15dd5896e083d34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.295054 1511 raft_consensus.cc:3060] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:10.295364 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 59956dad06064024ab171631ef6c1dca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.295636 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet d2c92975cf66441f9ac90c19772d3db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.295835 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4421b15a60d4256aeaa894bb1e4f99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.299322 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 839cef50795840fab564b94d88565f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.302228 1774 ts_tablet_manager.cc:1916] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.302976 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 60f591124dbb444fa7a5b7f408d575df (DEFAULT_TABLE table=default.Table_7065c04cd8e6495180d009455573cde0 [id=06ec4a8cd42a46cab05cbf63a1f6892d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.303256 1472 tablet_service.cc:1552] Processing DeleteTablet for tablet 2087e295ad4b4daa8eeebd49b43e08c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.304122 1476 tablet_service.cc:1505] Processing CreateTablet for tablet b24a3a820c9b496dbf6955b399c6ea8c (DEFAULT_TABLE table=default.Table_434a6ef23bea4f3991418b6ed2d9f635 [id=f24ffafc3bb24a57a21ec7e070a46d03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.322419 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de2ebb79d4fe4a1bbac372bb26f370ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.328081 1776 ts_tablet_manager.cc:1950] T 3754540ae21e4c65b5af3d0aa3f9189f P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:10.329527 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 576) took 2386 ms. Trace:
I20251024 19:48:10.330256 1767 tablet_bootstrap.cc:492] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:10.332957 1767 ts_tablet_manager.cc:1403] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.810s user 0.012s sys 0.024s
I20251024 19:48:10.332500 1776 rpcz_store.cc:276] 1024 19:48:07.942894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.943881 (+ 987us) service_pool.cc:225] Handling call
1024 19:48:10.177758 (+2233877us) ts_tablet_manager.cc:1151] Deleting tablet 3754540ae21e4c65b5af3d0aa3f9189f
1024 19:48:10.177954 (+ 196us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.277219 (+ 99265us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.283670 (+ 6451us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.329507 (+ 45837us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2168714}
I20251024 19:48:10.334578 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 3754540ae21e4c65b5af3d0aa3f9189f (table default.Table_eb1e63c7ef2244259c8a64fd4fe3859b [id=fec3ddedec0b44db80b30c899043bc15]) successfully deleted
I20251024 19:48:10.335251 1767 raft_consensus.cc:359] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.335924 1767 raft_consensus.cc:385] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251024 19:48:10.336647 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 65424b00ab9c454f9584048caba83e6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65424b00ab9c454f9584048caba83e6a already in progress: creating tablet
I20251024 19:48:10.336828 1767 raft_consensus.cc:740] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
W20251024 19:48:10.336913 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7eb62791100f477784ab315923b6ceb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7eb62791100f477784ab315923b6ceb6 already in progress: creating tablet
I20251024 19:48:10.335810 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 579) took 2323 ms. Trace:
I20251024 19:48:10.337093 1776 rpcz_store.cc:276] 1024 19:48:08.012519 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.012903 (+ 384us) service_pool.cc:225] Handling call
1024 19:48:10.335657 (+2322754us) ts_tablet_manager.cc:1151] Deleting tablet bc61fded84b24e00b299e5d342076708
1024 19:48:10.335671 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.335790 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2288206}
W20251024 19:48:10.337688 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet bc61fded84b24e00b299e5d342076708 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc61fded84b24e00b299e5d342076708 already in progress: creating tablet
I20251024 19:48:10.338150 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 580) took 2321 ms. Trace:
I20251024 19:48:10.338703 1776 rpcz_store.cc:276] 1024 19:48:08.016552 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.016692 (+ 140us) service_pool.cc:225] Handling call
1024 19:48:10.338018 (+2321326us) ts_tablet_manager.cc:1151] Deleting tablet 8af55c71122f4712ba1050e42f01ed83
1024 19:48:10.338031 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.338130 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2289134}
W20251024 19:48:10.339423 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 8af55c71122f4712ba1050e42f01ed83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8af55c71122f4712ba1050e42f01ed83 already in progress: creating tablet
I20251024 19:48:10.339865 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 581) took 2293 ms. Trace:
I20251024 19:48:10.341737 1776 rpcz_store.cc:276] 1024 19:48:08.046406 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.046706 (+ 300us) service_pool.cc:225] Handling call
1024 19:48:10.339739 (+2293033us) ts_tablet_manager.cc:1151] Deleting tablet c4d261f5f55949159aa1b72797e20fd5
1024 19:48:10.339752 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.339846 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2290312}
W20251024 19:48:10.342455 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet c4d261f5f55949159aa1b72797e20fd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4d261f5f55949159aa1b72797e20fd5 already in progress: creating tablet
I20251024 19:48:10.338505 1767 consensus_queue.cc:260] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.345428 1767 ts_tablet_manager.cc:1434] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251024 19:48:10.346098 1767 tablet_bootstrap.cc:492] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:48:10.347026 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 83b91a5d764449d5b82ef6d1ee0360f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83b91a5d764449d5b82ef6d1ee0360f5 already in progress: creating tablet
W20251024 19:48:10.347258 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet f88390f2534e4799a6eaa56aa4245e97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f88390f2534e4799a6eaa56aa4245e97 already in progress: creating tablet
W20251024 19:48:10.347450 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7deb97be82964ea0b2fe9caa867a6460 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7deb97be82964ea0b2fe9caa867a6460 already in progress: creating tablet
I20251024 19:48:10.346513 1586 leader_election.cc:304] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:10.352372 2501 raft_consensus.cc:2804] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:10.352419 1586 leader_election.cc:304] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:10.352468 2501 raft_consensus.cc:493] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:10.352528 2501 raft_consensus.cc:3060] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:10.353446 1586 leader_election.cc:343] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.353674 2584 raft_consensus.cc:2804] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:10.353739 2584 raft_consensus.cc:493] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251024 19:48:10.353785 1586 leader_election.cc:343] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:10.354053 1586 leader_election.cc:343] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.353798 2584 raft_consensus.cc:3060] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:10.354403 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7503b22d9e12481b9af96385b9d76e50" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:10.354779 1586 leader_election.cc:343] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:10.355044 1586 leader_election.cc:343] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.355764 1774 ts_tablet_manager.cc:1929] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:10.355885 1774 log.cc:1199] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/0a12e415590a4c71aab4ad7d30412c83
I20251024 19:48:10.360316 2501 raft_consensus.cc:515] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.360461 1774 ts_tablet_manager.cc:1950] T 0a12e415590a4c71aab4ad7d30412c83 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:10.360875 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 485) took 3022 ms. Trace:
I20251024 19:48:10.360920 2501 leader_election.cc:290] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:10.362149 2584 raft_consensus.cc:515] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.362679 2584 leader_election.cc:290] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:10.372707 1767 tablet_bootstrap.cc:654] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.373989 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3259823b622461fa823841925b3d203" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:10.374696 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2037fcca87b94b5fb31639b4d07307b0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:10.377209 1472 tablet_service.cc:1505] Processing CreateTablet for tablet a5442a5604134c4d86cfe6ca730146f0 (DEFAULT_TABLE table=default.Table_733a79b22a954b088ad061fc232e3008 [id=6c5a74f370f14c6588505015c6136507]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.377687 1767 tablet_bootstrap.cc:492] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:10.377810 1767 ts_tablet_manager.cc:1403] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20251024 19:48:10.304606 1474 tablet_service.cc:1505] Processing CreateTablet for tablet bcdd47a64e2142cda1f2d20afebd4ce5 (DEFAULT_TABLE table=default.Table_56c54a42391a4397bdbcc0964ed0f9e7 [id=01759ac361e14c15a31441a0ef8db7db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.378165 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9198321bcc9e4e47a4c2a5442b6bc850. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.378512 1767 raft_consensus.cc:359] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.378669 1767 raft_consensus.cc:385] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.378716 1767 raft_consensus.cc:740] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.378917 1767 consensus_queue.cc:260] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.381968 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f591124dbb444fa7a5b7f408d575df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.382736 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcdd47a64e2142cda1f2d20afebd4ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.385862 1767 ts_tablet_manager.cc:1434] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20251024 19:48:10.388543 1586 leader_election.cc:343] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:10.388729 1586 leader_election.cc:343] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.388984 2584 consensus_queue.cc:1048] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:10.350698 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 583) took 2241 ms. Trace:
I20251024 19:48:10.386626 1767 tablet_bootstrap.cc:492] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:10.389442 1776 rpcz_store.cc:276] 1024 19:48:08.109526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.109601 (+ 75us) service_pool.cc:225] Handling call
1024 19:48:10.350579 (+2240978us) ts_tablet_manager.cc:1151] Deleting tablet c1cdd5ec32ca49c1bfafaf3a24050e89
1024 19:48:10.350590 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.350679 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2240756}
I20251024 19:48:10.392472 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 582) took 2294 ms. Trace:
W20251024 19:48:10.392474 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1cdd5ec32ca49c1bfafaf3a24050e89 already in progress: creating tablet
I20251024 19:48:10.392635 1776 rpcz_store.cc:276] 1024 19:48:08.098089 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.102753 (+ 4664us) service_pool.cc:225] Handling call
1024 19:48:10.392323 (+2289570us) ts_tablet_manager.cc:1151] Deleting tablet 55986287a8bb40c8bb28e536fb22fb27
1024 19:48:10.392334 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.392454 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2277176}
I20251024 19:48:10.393306 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 584) took 2235 ms. Trace:
I20251024 19:48:10.393452 1776 rpcz_store.cc:276] 1024 19:48:08.158368 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.172104 (+ 13736us) service_pool.cc:225] Handling call
1024 19:48:10.393163 (+2221059us) ts_tablet_manager.cc:1151] Deleting tablet 57f522a825264ea7b215304dbb7aa2d5
1024 19:48:10.393185 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.393285 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2220696}
W20251024 19:48:10.393916 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 57f522a825264ea7b215304dbb7aa2d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57f522a825264ea7b215304dbb7aa2d5 already in progress: creating tablet
I20251024 19:48:10.394059 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 604) took 1627 ms. Trace:
I20251024 19:48:10.394212 1776 rpcz_store.cc:276] 1024 19:48:08.766099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.766145 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:10.393893 (+1627748us) ts_tablet_manager.cc:1151] Deleting tablet 4c54e569156042fca5e9ab0feaedf632
1024 19:48:10.393905 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.393980 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1627496}
W20251024 19:48:10.394714 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4c54e569156042fca5e9ab0feaedf632 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c54e569156042fca5e9ab0feaedf632 already in progress: creating tablet
I20251024 19:48:10.394884 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 605) took 1621 ms. Trace:
I20251024 19:48:10.394996 1776 rpcz_store.cc:276] 1024 19:48:08.773929 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.774315 (+ 386us) service_pool.cc:225] Handling call
1024 19:48:10.394700 (+1620385us) ts_tablet_manager.cc:1151] Deleting tablet b240965605904533bd2af99f61aaef26
1024 19:48:10.394711 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.394863 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1620148}
W20251024 19:48:10.395485 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b240965605904533bd2af99f61aaef26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b240965605904533bd2af99f61aaef26 already in progress: creating tablet
I20251024 19:48:10.395545 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 606) took 1621 ms. Trace:
I20251024 19:48:10.395761 1776 rpcz_store.cc:276] 1024 19:48:08.774126 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.774595 (+ 469us) service_pool.cc:225] Handling call
1024 19:48:10.395444 (+1620849us) ts_tablet_manager.cc:1151] Deleting tablet bbbc3f1886e744ab9450a0f5776721d9
1024 19:48:10.395455 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.395531 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1620658}
W20251024 19:48:10.396178 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet bbbc3f1886e744ab9450a0f5776721d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbbc3f1886e744ab9450a0f5776721d9 already in progress: creating tablet
I20251024 19:48:10.396347 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 607) took 1622 ms. Trace:
I20251024 19:48:10.396500 1776 rpcz_store.cc:276] 1024 19:48:08.774259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.779439 (+ 5180us) service_pool.cc:225] Handling call
1024 19:48:10.396167 (+1616728us) ts_tablet_manager.cc:1151] Deleting tablet f51c56337ebb450aaf62dcc81ef63ae8
1024 19:48:10.396177 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.396327 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600857}
W20251024 19:48:10.396991 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f51c56337ebb450aaf62dcc81ef63ae8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f51c56337ebb450aaf62dcc81ef63ae8 already in progress: creating tablet
I20251024 19:48:10.397058 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 608) took 1595 ms. Trace:
I20251024 19:48:10.397924 1767 tablet_bootstrap.cc:654] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.397950 1776 rpcz_store.cc:276] 1024 19:48:08.801927 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.802518 (+ 591us) service_pool.cc:225] Handling call
1024 19:48:10.396962 (+1594444us) ts_tablet_manager.cc:1151] Deleting tablet 98dd8bacd8374f469799872859db4206
1024 19:48:10.396973 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.397044 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1592628}
W20251024 19:48:10.401952 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 98dd8bacd8374f469799872859db4206 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98dd8bacd8374f469799872859db4206 already in progress: creating tablet
I20251024 19:48:10.402415 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 609) took 1475 ms. Trace:
I20251024 19:48:10.402814 1776 rpcz_store.cc:276] 1024 19:48:08.926842 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.927186 (+ 344us) service_pool.cc:225] Handling call
1024 19:48:10.402322 (+1475136us) ts_tablet_manager.cc:1151] Deleting tablet 9fffc42ddda14bd28800917ff2f0ced5
1024 19:48:10.402333 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.402398 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1474821}
W20251024 19:48:10.403664 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9fffc42ddda14bd28800917ff2f0ced5 because the tablet was not found. No further retry: Not found: Tablet not found: 9fffc42ddda14bd28800917ff2f0ced5
I20251024 19:48:10.404099 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 610) took 1477 ms. Trace:
I20251024 19:48:10.406136 1776 rpcz_store.cc:276] 1024 19:48:08.926978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.927543 (+ 565us) service_pool.cc:225] Handling call
1024 19:48:10.404021 (+1476478us) ts_tablet_manager.cc:1151] Deleting tablet b6661b9e9a174849b6093d09cd8c1251
1024 19:48:10.404030 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.404084 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1476314}
W20251024 19:48:10.406898 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b6661b9e9a174849b6093d09cd8c1251 because the tablet was not found. No further retry: Not found: Tablet not found: b6661b9e9a174849b6093d09cd8c1251
I20251024 19:48:10.407563 1776 tablet_replica.cc:333] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:10.408366 1776 raft_consensus.cc:2243] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:10.408864 1776 pending_rounds.cc:70] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:10.410091 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b24a3a820c9b496dbf6955b399c6ea8c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:10.412245 1473 consensus_meta.cc:287] T 1e6a588092a44688bc96f16ac00e39f9 P 10a9536207e343f188e0ef709081d04e: Time spent flushing consensus metadata: real 0.615s user 0.004s sys 0.000s
I20251024 19:48:10.412618 1473 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46462 (request call id 608) took 1209 ms. Trace:
I20251024 19:48:10.413406 1776 pending_rounds.cc:77] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429348602441728 op_type: NO_OP noop_request { }
I20251024 19:48:10.414462 1776 raft_consensus.cc:2889] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:10.414929 1776 raft_consensus.cc:2272] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.412724 1473 rpcz_store.cc:276] 1024 19:48:09.202959 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.203049 (+ 90us) service_pool.cc:225] Handling call
1024 19:48:09.344478 (+141429us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1024 19:48:09.344521 (+ 43us) ts_tablet_manager.cc:682] Creating new metadata...
1024 19:48:09.797246 (+452725us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.412425 (+615179us) tablet_replica.cc:206] Creating consensus instance
1024 19:48:10.412600 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251024 19:48:10.414976 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7503b22d9e12481b9af96385b9d76e50" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:10.417409 1509 raft_consensus.cc:3060] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:10.418414 1473 tablet_service.cc:1505] Processing CreateTablet for tablet af5414e8275845f58717f19fa1b900ed (DEFAULT_TABLE table=default.Table_04c43324857b4aa7b8a4c4f6eb0769f3 [id=3d2816d24b694e609b335150c39d4192]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.422377 1774 rpcz_store.cc:276] 1024 19:48:07.338677 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.338790 (+ 113us) service_pool.cc:225] Handling call
1024 19:48:10.292050 (+2953260us) ts_tablet_manager.cc:1151] Deleting tablet 0a12e415590a4c71aab4ad7d30412c83
1024 19:48:10.292060 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.343596 (+ 51536us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.355640 (+ 12044us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.360831 (+ 5191us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2947826}
I20251024 19:48:10.422930 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 484) took 3099 ms. Trace:
I20251024 19:48:10.423038 1774 rpcz_store.cc:276] 1024 19:48:07.323447 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.324128 (+ 681us) service_pool.cc:225] Handling call
1024 19:48:10.422777 (+3098649us) ts_tablet_manager.cc:1151] Deleting tablet fc7ccf1495924782b9b6ac1173bc73a1
1024 19:48:10.422790 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.422910 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3078366}
I20251024 19:48:10.423429 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 486) took 3061 ms. Trace:
I20251024 19:48:10.423511 1774 rpcz_store.cc:276] 1024 19:48:07.362458 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.362629 (+ 171us) service_pool.cc:225] Handling call
1024 19:48:10.423303 (+3060674us) ts_tablet_manager.cc:1151] Deleting tablet 0266cef4e2034303978d7456c47ea98c
1024 19:48:10.423315 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.423398 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3058683}
I20251024 19:48:10.423823 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 487) took 3045 ms. Trace:
I20251024 19:48:10.423899 1774 rpcz_store.cc:276] 1024 19:48:07.378536 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.379036 (+ 500us) service_pool.cc:225] Handling call
1024 19:48:10.423723 (+3044687us) ts_tablet_manager.cc:1151] Deleting tablet 4193750331874d7ba9bbe6279c3a8e37
1024 19:48:10.423733 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.423809 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3037364}
I20251024 19:48:10.424206 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 488) took 3008 ms. Trace:
I20251024 19:48:10.424285 1774 rpcz_store.cc:276] 1024 19:48:07.416239 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.416315 (+ 76us) service_pool.cc:225] Handling call
1024 19:48:10.424111 (+3007796us) ts_tablet_manager.cc:1151] Deleting tablet 7712392f200842c4b728e019d6a3cfb0
1024 19:48:10.424119 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.424190 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3007566}
I20251024 19:48:10.424598 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 489) took 3006 ms. Trace:
I20251024 19:48:10.424681 1774 rpcz_store.cc:276] 1024 19:48:07.418249 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.418739 (+ 490us) service_pool.cc:225] Handling call
1024 19:48:10.424494 (+3005755us) ts_tablet_manager.cc:1151] Deleting tablet d4b3a50db1714eb38cc0ea03f612fc95
1024 19:48:10.424502 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.424583 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3005569}
I20251024 19:48:10.427570 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5442a5604134c4d86cfe6ca730146f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.428850 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af5414e8275845f58717f19fa1b900ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.430440 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3259823b622461fa823841925b3d203" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:10.418776 2610 tablet.cc:1722] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:10.430691 1505 raft_consensus.cc:3060] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:10.431164 1481 consensus_meta.cc:287] T 2bb5f84c1f104b338be71cb55ae77929 P 10a9536207e343f188e0ef709081d04e: Time spent flushing consensus metadata: real 0.723s user 0.004s sys 0.000s
I20251024 19:48:10.433638 1776 ts_tablet_manager.cc:1916] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.447372 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 490) took 3005 ms. Trace:
I20251024 19:48:10.448146 1774 rpcz_store.cc:276] 1024 19:48:07.442164 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.442226 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:10.424892 (+2982666us) ts_tablet_manager.cc:1151] Deleting tablet ce8c179e98944df0b510a0e8daf3920f
1024 19:48:10.424900 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.447352 (+ 22452us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2982431}
I20251024 19:48:10.449839 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 492) took 2942 ms. Trace:
I20251024 19:48:10.450495 1481 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46462 (request call id 618) took 1095 ms. Trace:
I20251024 19:48:10.450609 1481 rpcz_store.cc:276] 1024 19:48:09.355100 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.355305 (+ 205us) service_pool.cc:225] Handling call
1024 19:48:09.355751 (+ 446us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1024 19:48:09.355778 (+ 27us) ts_tablet_manager.cc:682] Creating new metadata...
1024 19:48:09.690460 (+334682us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.433680 (+743220us) tablet_replica.cc:206] Creating consensus instance
1024 19:48:10.450459 (+ 16779us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251024 19:48:10.450664 1774 rpcz_store.cc:276] 1024 19:48:07.507092 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.507967 (+ 875us) service_pool.cc:225] Handling call
1024 19:48:10.449724 (+2941757us) ts_tablet_manager.cc:1151] Deleting tablet 4cac515c0a00403fb680454c1721be0a
1024 19:48:10.449734 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.449822 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2939885}
I20251024 19:48:10.430766 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2037fcca87b94b5fb31639b4d07307b0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:10.452118 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 497) took 2867 ms. Trace:
I20251024 19:48:10.452412 1481 tablet_service.cc:1505] Processing CreateTablet for tablet 584cbe0c396841f794e665aae3ebbd91 (DEFAULT_TABLE table=default.Table_35c6a6e931bc4596a02d717ead9865ce [id=2cf6b584c9b54f9fad960f515568b21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.452543 1774 rpcz_store.cc:276] 1024 19:48:07.584679 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.594717 (+ 10038us) service_pool.cc:225] Handling call
1024 19:48:10.451746 (+2857029us) ts_tablet_manager.cc:1151] Deleting tablet 1295c99836d14a1cb0002e6c6aaee1eb
1024 19:48:10.451756 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.452099 (+ 343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2856807}
I20251024 19:48:10.453449 1774 tablet_replica.cc:333] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:10.453533 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 0a12e415590a4c71aab4ad7d30412c83 (table default.Table_894d6739340148a995dd849a205c19c3 [id=18ec057e920e4c21ae8cbe298058b6f3]) successfully deleted
W20251024 19:48:10.453908 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet fc7ccf1495924782b9b6ac1173bc73a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc7ccf1495924782b9b6ac1173bc73a1 already in progress: creating tablet
W20251024 19:48:10.454267 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 0266cef4e2034303978d7456c47ea98c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0266cef4e2034303978d7456c47ea98c already in progress: creating tablet
W20251024 19:48:10.454519 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4193750331874d7ba9bbe6279c3a8e37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4193750331874d7ba9bbe6279c3a8e37 already in progress: creating tablet
I20251024 19:48:10.454322 1774 raft_consensus.cc:2243] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:10.454830 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7712392f200842c4b728e019d6a3cfb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7712392f200842c4b728e019d6a3cfb0 already in progress: creating tablet
I20251024 19:48:10.455123 1774 raft_consensus.cc:2272] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:10.455174 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet d4b3a50db1714eb38cc0ea03f612fc95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4b3a50db1714eb38cc0ea03f612fc95 already in progress: creating tablet
I20251024 19:48:10.454511 1504 raft_consensus.cc:3060] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:10.455690 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet ce8c179e98944df0b510a0e8daf3920f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce8c179e98944df0b510a0e8daf3920f already in progress: creating tablet
W20251024 19:48:10.456166 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4cac515c0a00403fb680454c1721be0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cac515c0a00403fb680454c1721be0a already in progress: creating tablet
W20251024 19:48:10.456597 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 1295c99836d14a1cb0002e6c6aaee1eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1295c99836d14a1cb0002e6c6aaee1eb already in progress: creating tablet
I20251024 19:48:10.457379 1774 ts_tablet_manager.cc:1916] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.455261 1481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 584cbe0c396841f794e665aae3ebbd91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.462827 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:54172 (request call id 126) took 1034 ms. Trace:
I20251024 19:48:10.462975 1332 rpcz_store.cc:276] 1024 19:48:09.429106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.429189 (+ 83us) service_pool.cc:225] Handling call
1024 19:48:09.572032 (+142843us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x6160001a1d54. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:10.462804 (+890772us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":314148224}
I20251024 19:48:10.465117 1776 ts_tablet_manager.cc:1929] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:10.467957 1776 log.cc:1199] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/ca0527af718f494a8bbcc66e7b325bed
I20251024 19:48:10.468348 1776 ts_tablet_manager.cc:1950] T ca0527af718f494a8bbcc66e7b325bed P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:10.474532 1509 raft_consensus.cc:2468] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:10.475314 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 6b9d7c4ecafe40d698187564201582bb (DEFAULT_TABLE table=default.Table_cfeecdf5f1bd4eb099bf97e5713a4d41 [id=34657e83e42f410ca0d4ed4e036d15df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.469322 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 611) took 1542 ms. Trace:
I20251024 19:48:10.478404 1776 rpcz_store.cc:276] 1024 19:48:08.927124 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.927742 (+ 618us) service_pool.cc:225] Handling call
1024 19:48:10.407514 (+1479772us) ts_tablet_manager.cc:1151] Deleting tablet ca0527af718f494a8bbcc66e7b325bed
1024 19:48:10.407523 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.440917 (+ 33394us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.465007 (+ 24090us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.469301 (+ 4294us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1479584}
I20251024 19:48:10.479223 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet ca0527af718f494a8bbcc66e7b325bed (table default.Table_51982d999b0847ec87fc62a473782a5c [id=9a05bcf7ad5d430c9ef6d6e18a27808e]) successfully deleted
I20251024 19:48:10.479607 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 612) took 1532 ms. Trace:
W20251024 19:48:10.476428 1447 leader_election.cc:343] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.480055 1776 rpcz_store.cc:276] 1024 19:48:08.947035 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.947111 (+ 76us) service_pool.cc:225] Handling call
1024 19:48:10.479280 (+1532169us) ts_tablet_manager.cc:1151] Deleting tablet 23d4dd0b42d343cb9354ac698a8c761f
1024 19:48:10.479292 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.479587 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1531939}
I20251024 19:48:10.480876 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 616) took 1434 ms. Trace:
W20251024 19:48:10.480859 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 23d4dd0b42d343cb9354ac698a8c761f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23d4dd0b42d343cb9354ac698a8c761f already in progress: creating tablet
I20251024 19:48:10.481500 1776 rpcz_store.cc:276] 1024 19:48:09.045991 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.046725 (+ 734us) service_pool.cc:225] Handling call
1024 19:48:10.480772 (+1434047us) ts_tablet_manager.cc:1151] Deleting tablet 2bb5f84c1f104b338be71cb55ae77929
1024 19:48:10.480782 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.480862 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1433706}
W20251024 19:48:10.482250 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2bb5f84c1f104b338be71cb55ae77929 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bb5f84c1f104b338be71cb55ae77929 already in progress: creating tablet
I20251024 19:48:10.482287 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 617) took 1436 ms. Trace:
I20251024 19:48:10.482844 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b9d7c4ecafe40d698187564201582bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.482959 1774 ts_tablet_manager.cc:1929] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:10.483076 1774 log.cc:1199] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/a76d61c3464b41519d7657796bd853ff
I20251024 19:48:10.483412 1774 ts_tablet_manager.cc:1950] T a76d61c3464b41519d7657796bd853ff P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:10.483789 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 499) took 2838 ms. Trace:
I20251024 19:48:10.483886 1774 rpcz_store.cc:276] 1024 19:48:07.645247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.645304 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:10.453400 (+2808096us) ts_tablet_manager.cc:1151] Deleting tablet a76d61c3464b41519d7657796bd853ff
1024 19:48:10.453409 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.466574 (+ 13165us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.482844 (+ 16270us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.483770 (+ 926us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2805807}
I20251024 19:48:10.479820 1447 leader_election.cc:304] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:10.484382 1776 rpcz_store.cc:276] 1024 19:48:09.046340 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.047106 (+ 766us) service_pool.cc:225] Handling call
1024 19:48:10.482179 (+1435073us) ts_tablet_manager.cc:1151] Deleting tablet e68997931ede4b6398570a7d08edaf4d
1024 19:48:10.482189 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.482272 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1434726}
W20251024 19:48:10.484958 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet e68997931ede4b6398570a7d08edaf4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e68997931ede4b6398570a7d08edaf4d already in progress: creating tablet
I20251024 19:48:10.485453 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 618) took 1438 ms. Trace:
I20251024 19:48:10.486430 1504 raft_consensus.cc:2468] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:10.486799 1776 rpcz_store.cc:276] 1024 19:48:09.046643 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.047498 (+ 855us) service_pool.cc:225] Handling call
1024 19:48:10.485319 (+1437821us) ts_tablet_manager.cc:1151] Deleting tablet 95ae017a2e084ccabfa69c9a5befe0e7
1024 19:48:10.485333 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.485433 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1437489}
W20251024 19:48:10.489399 1447 consensus_peers.cc:597] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc1b6f0a377945089c50012f74d38564. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:10.489806 1447 consensus_peers.cc:597] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aebf76d4a66947ed9756643740418798. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:10.490252 2334 raft_consensus.cc:2749] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:10.491209 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 95ae017a2e084ccabfa69c9a5befe0e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95ae017a2e084ccabfa69c9a5befe0e7 already in progress: creating tablet
I20251024 19:48:10.491273 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 620) took 1418 ms. Trace:
I20251024 19:48:10.491847 1776 rpcz_store.cc:276] 1024 19:48:09.072558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.073200 (+ 642us) service_pool.cc:225] Handling call
1024 19:48:10.491134 (+1417934us) ts_tablet_manager.cc:1151] Deleting tablet 1e6a588092a44688bc96f16ac00e39f9
1024 19:48:10.491147 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.491251 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1417599}
I20251024 19:48:10.493157 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 520) took 2470 ms. Trace:
I20251024 19:48:10.493625 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 622) took 1411 ms. Trace:
W20251024 19:48:10.493634 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 1e6a588092a44688bc96f16ac00e39f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e6a588092a44688bc96f16ac00e39f9 already in progress: creating tablet
I20251024 19:48:10.493815 1776 rpcz_store.cc:276] 1024 19:48:09.082217 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.082719 (+ 502us) service_pool.cc:225] Handling call
1024 19:48:10.493519 (+1410800us) ts_tablet_manager.cc:1151] Deleting tablet aebf76d4a66947ed9756643740418798
1024 19:48:10.493529 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.493611 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1410469}
I20251024 19:48:10.494028 1774 rpcz_store.cc:276] 1024 19:48:08.023179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.023218 (+ 39us) service_pool.cc:225] Handling call
1024 19:48:10.493025 (+2469807us) ts_tablet_manager.cc:1151] Deleting tablet 8af55c71122f4712ba1050e42f01ed83
1024 19:48:10.493037 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.493138 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2469549}
I20251024 19:48:10.494684 1776 tablet_replica.cc:333] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:48:10.494750 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet aebf76d4a66947ed9756643740418798 because the tablet was not found. No further retry: Not found: Tablet not found: aebf76d4a66947ed9756643740418798
W20251024 19:48:10.495020 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 8af55c71122f4712ba1050e42f01ed83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8af55c71122f4712ba1050e42f01ed83 already in progress: creating tablet
I20251024 19:48:10.495296 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 521) took 2461 ms. Trace:
I20251024 19:48:10.495544 1776 raft_consensus.cc:2243] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:10.495759 1774 rpcz_store.cc:276] 1024 19:48:08.034113 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.034493 (+ 380us) service_pool.cc:225] Handling call
1024 19:48:10.495179 (+2460686us) ts_tablet_manager.cc:1151] Deleting tablet c4d261f5f55949159aa1b72797e20fd5
1024 19:48:10.495190 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.495279 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2460452}
I20251024 19:48:10.496110 1776 pending_rounds.cc:70] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:10.496786 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 519) took 2473 ms. Trace:
W20251024 19:48:10.496474 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet c4d261f5f55949159aa1b72797e20fd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4d261f5f55949159aa1b72797e20fd5 already in progress: creating tablet
I20251024 19:48:10.497030 1776 pending_rounds.cc:77] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429343402405888 op_type: NO_OP noop_request { }
I20251024 19:48:10.497290 1776 raft_consensus.cc:2889] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:10.497417 1774 rpcz_store.cc:276] 1024 19:48:08.023024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.023107 (+ 83us) service_pool.cc:225] Handling call
1024 19:48:10.496641 (+2473534us) ts_tablet_manager.cc:1151] Deleting tablet bc61fded84b24e00b299e5d342076708
1024 19:48:10.496661 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.496766 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2450300}
I20251024 19:48:10.497697 1776 raft_consensus.cc:2272] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:10.498565 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet bc61fded84b24e00b299e5d342076708 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc61fded84b24e00b299e5d342076708 already in progress: creating tablet
I20251024 19:48:10.497946 1768 tablet_bootstrap.cc:492] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:10.499006 1768 ts_tablet_manager.cc:1403] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.381s user 0.016s sys 0.041s
I20251024 19:48:10.499218 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 522) took 2399 ms. Trace:
I20251024 19:48:10.499336 1774 rpcz_store.cc:276] 1024 19:48:08.099573 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.099640 (+ 67us) service_pool.cc:225] Handling call
1024 19:48:10.498858 (+2399218us) ts_tablet_manager.cc:1151] Deleting tablet 55986287a8bb40c8bb28e536fb22fb27
1024 19:48:10.498867 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.499203 (+ 336us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2398852}
I20251024 19:48:10.499698 1776 ts_tablet_manager.cc:1916] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.499786 1768 raft_consensus.cc:359] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.499959 1768 raft_consensus.cc:385] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.500016 1768 raft_consensus.cc:740] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.500224 1768 consensus_queue.cc:260] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.500684 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 523) took 2387 ms. Trace:
I20251024 19:48:10.501454 1774 rpcz_store.cc:276] 1024 19:48:08.113521 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.113597 (+ 76us) service_pool.cc:225] Handling call
1024 19:48:10.500510 (+2386913us) ts_tablet_manager.cc:1151] Deleting tablet c1cdd5ec32ca49c1bfafaf3a24050e89
1024 19:48:10.500520 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.500667 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2376433}
I20251024 19:48:10.502023 1768 ts_tablet_manager.cc:1434] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251024 19:48:10.502002 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1cdd5ec32ca49c1bfafaf3a24050e89 already in progress: creating tablet
I20251024 19:48:10.502461 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 524) took 2315 ms. Trace:
I20251024 19:48:10.505254 1774 rpcz_store.cc:276] 1024 19:48:08.186850 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.187002 (+ 152us) service_pool.cc:225] Handling call
1024 19:48:10.502343 (+2315341us) ts_tablet_manager.cc:1151] Deleting tablet 57f522a825264ea7b215304dbb7aa2d5
1024 19:48:10.502352 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.502444 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2308449}
W20251024 19:48:10.505844 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 57f522a825264ea7b215304dbb7aa2d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57f522a825264ea7b215304dbb7aa2d5 already in progress: creating tablet
I20251024 19:48:10.506181 1774 tablet_replica.cc:333] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:10.506392 1774 raft_consensus.cc:2243] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:10.506564 1774 pending_rounds.cc:70] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:10.506640 1774 pending_rounds.cc:77] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429342251991040 op_type: NO_OP noop_request { }
I20251024 19:48:10.506796 1774 raft_consensus.cc:2889] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:10.506933 1774 raft_consensus.cc:2272] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.507514 1505 raft_consensus.cc:2468] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:10.503180 1768 tablet_bootstrap.cc:492] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:10.508181 1774 ts_tablet_manager.cc:1916] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.509171 1768 tablet_bootstrap.cc:654] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.516215 1776 ts_tablet_manager.cc:1929] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:10.516377 1511 raft_consensus.cc:2468] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:10.519622 1776 log.cc:1199] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/5368789df6024d7daa9ccc2257137d58
I20251024 19:48:10.520118 1776 ts_tablet_manager.cc:1950] T 5368789df6024d7daa9ccc2257137d58 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:10.520985 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 623) took 1438 ms. Trace:
I20251024 19:48:10.521826 1768 tablet_bootstrap.cc:492] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:10.521939 1776 rpcz_store.cc:276] 1024 19:48:09.082486 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.083218 (+ 732us) service_pool.cc:225] Handling call
1024 19:48:10.494617 (+1411399us) ts_tablet_manager.cc:1151] Deleting tablet 5368789df6024d7daa9ccc2257137d58
1024 19:48:10.494628 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.512663 (+ 18035us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.516116 (+ 3453us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.520964 (+ 4848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1411065}
I20251024 19:48:10.522200 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4 (DEFAULT_TABLE table=default.Table_d101352faa2b4f4b9b983143a5142043 [id=037cdc76ba24493595393d134223c759]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.522745 1776 tablet_replica.cc:333] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:10.522962 1768 ts_tablet_manager.cc:1403] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.002s
I20251024 19:48:10.523258 1776 raft_consensus.cc:2243] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:10.524240 1768 raft_consensus.cc:359] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.524590 1776 raft_consensus.cc:2272] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.525053 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 5368789df6024d7daa9ccc2257137d58 (table default.Table_a81e7aae0a7a4f3db5c84efc4b16f873 [id=16bcec63f27d4dcd826ef11d45d58400]) successfully deleted
I20251024 19:48:10.525534 1776 ts_tablet_manager.cc:1916] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.538940 1768 raft_consensus.cc:385] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.539464 1768 raft_consensus.cc:740] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.544190 1187 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet d692e1988d13460fb1d54e930e17fc9d (table default.Table_b09229e0e1254adeacfed952befaa65a [id=67a0be12c85648bdb4515a096db21d3b]). Expected version 1 got 0
I20251024 19:48:10.544199 1768 consensus_queue.cc:260] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.544734 1768 ts_tablet_manager.cc:1434] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20251024 19:48:10.545006 1768 tablet_bootstrap.cc:492] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:10.522509 1774 ts_tablet_manager.cc:1929] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:10.547022 1774 log.cc:1199] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/3754540ae21e4c65b5af3d0aa3f9189f
I20251024 19:48:10.548520 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 069091f1c238487580baab38af6a70e2
I20251024 19:48:10.550102 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ae0d26f010bd46d3a05ac50f966d5769
I20251024 19:48:10.551074 1369 raft_consensus.cc:3060] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:10.551158 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 55aa786b9f6847baa265802cc28b2b80
I20251024 19:48:10.551698 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet 2037fcca87b94b5fb31639b4d07307b0 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:10.553614 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 069091f1c238487580baab38af6a70e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:10.553750 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet ae0d26f010bd46d3a05ac50f966d5769 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:10.553925 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 55aa786b9f6847baa265802cc28b2b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:10.554486 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet ae0d26f010bd46d3a05ac50f966d5769 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:10.554967 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 55aa786b9f6847baa265802cc28b2b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:10.556457 1774 ts_tablet_manager.cc:1950] T 3754540ae21e4c65b5af3d0aa3f9189f P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:10.556810 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 526) took 2028 ms. Trace:
I20251024 19:48:10.558705 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b1d45a27e5bd42e4a24d1a89f49f9221" } new_table_name: "default.Table_22b072e068244a37b59bbac02c3dfb75" modify_external_catalogs: true
I20251024 19:48:10.558940 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 27f1a5eccdf6430c9e220988043596b0
I20251024 19:48:10.559013 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 069091f1c238487580baab38af6a70e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:10.559653 1491 tablet_service.cc:1505] Processing CreateTablet for tablet e7255804242d424f8cbf89ef57d4ff9f (DEFAULT_TABLE table=default.Table_7b543dc2852946a78c82d145655196ae [id=191585bc444043ceb08ae1e5f7ee00a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.559665 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_077b6c98744f4a5baa42d5c4bd0da645"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:10.560200 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_077b6c98744f4a5baa42d5c4bd0da645 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:10.561980 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 27f1a5eccdf6430c9e220988043596b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:10.562300 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 3a712f52cd114e9bb07c2717e5d728fd (DEFAULT_TABLE table=default.Table_fa23a3c70c3247f9ac84e69f79a41640 [id=b4fdd246321146cdbf968d162389fe50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.563041 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_81a931b6624742858d05cfc15a78beed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:10.563488 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 27f1a5eccdf6430c9e220988043596b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:10.563639 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81a931b6624742858d05cfc15a78beed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:10.564321 1766 tablet_bootstrap.cc:492] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:10.565719 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 3bacc8601ca1437586bdc9a52e086299 (DEFAULT_TABLE table=default.Table_9fee2e99d42e4cccae5d6e4825802713 [id=63c6383bbaf7493b96eb6e7230695800]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.568876 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9ee642f4c0344db6b73e09e30ab6d351"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:10.569728 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 4846799df5b242ce8f19c034320c8a1d (DEFAULT_TABLE table=default.Table_b1d45a27e5bd42e4a24d1a89f49f9221 [id=fe6f3191d9fb4afbb64b4097912afc89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.570123 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:10.571377 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ee642f4c0344db6b73e09e30ab6d351 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:10.556905 1774 rpcz_store.cc:276] 1024 19:48:08.528295 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.537882 (+ 9587us) service_pool.cc:225] Handling call
1024 19:48:10.506137 (+1968255us) ts_tablet_manager.cc:1151] Deleting tablet 3754540ae21e4c65b5af3d0aa3f9189f
1024 19:48:10.506146 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.511462 (+ 5316us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.519081 (+ 7619us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.556795 (+ 37714us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1968007}
I20251024 19:48:10.573055 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 3754540ae21e4c65b5af3d0aa3f9189f (table default.Table_eb1e63c7ef2244259c8a64fd4fe3859b [id=fec3ddedec0b44db80b30c899043bc15]) successfully deleted
I20251024 19:48:10.573482 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 546) took 1794 ms. Trace:
I20251024 19:48:10.573701 1774 rpcz_store.cc:276] 1024 19:48:08.779063 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.783067 (+ 4004us) service_pool.cc:225] Handling call
1024 19:48:10.573353 (+1790286us) ts_tablet_manager.cc:1151] Deleting tablet b240965605904533bd2af99f61aaef26
1024 19:48:10.573363 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.573464 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1786613}
W20251024 19:48:10.574329 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b240965605904533bd2af99f61aaef26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b240965605904533bd2af99f61aaef26 already in progress: creating tablet
I20251024 19:48:10.575006 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 548) took 1795 ms. Trace:
I20251024 19:48:10.575124 1774 rpcz_store.cc:276] 1024 19:48:08.779364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.784204 (+ 4840us) service_pool.cc:225] Handling call
1024 19:48:10.574869 (+1790665us) ts_tablet_manager.cc:1151] Deleting tablet f51c56337ebb450aaf62dcc81ef63ae8
1024 19:48:10.574881 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.574989 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1776949}
W20251024 19:48:10.576445 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet f51c56337ebb450aaf62dcc81ef63ae8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f51c56337ebb450aaf62dcc81ef63ae8 already in progress: creating tablet
I20251024 19:48:10.580124 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 549) took 1790 ms. Trace:
I20251024 19:48:10.580246 1774 rpcz_store.cc:276] 1024 19:48:08.789714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.790000 (+ 286us) service_pool.cc:225] Handling call
1024 19:48:10.578912 (+1788912us) ts_tablet_manager.cc:1151] Deleting tablet 98dd8bacd8374f469799872859db4206
1024 19:48:10.580018 (+ 1106us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.580109 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1780156}
I20251024 19:48:10.580263 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 57e1e21beee44847b1ebeb746f02c372 (DEFAULT_TABLE table=default.Table_a6498b82a207414dafe80255fff08e8c [id=73c28b9621534cb896be0badbe3911be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.582585 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4846799df5b242ce8f19c034320c8a1d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:10.582617 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 98dd8bacd8374f469799872859db4206 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98dd8bacd8374f469799872859db4206 already in progress: creating tablet
I20251024 19:48:10.583164 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a712f52cd114e9bb07c2717e5d728fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.585395 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 547) took 1806 ms. Trace:
I20251024 19:48:10.585494 1774 rpcz_store.cc:276] 1024 19:48:08.779211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.786425 (+ 7214us) service_pool.cc:225] Handling call
1024 19:48:10.580564 (+1794139us) ts_tablet_manager.cc:1151] Deleting tablet bbbc3f1886e744ab9450a0f5776721d9
1024 19:48:10.585247 (+ 4683us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.585374 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1781010}
W20251024 19:48:10.586033 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet bbbc3f1886e744ab9450a0f5776721d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbbc3f1886e744ab9450a0f5776721d9 already in progress: creating tablet
I20251024 19:48:10.586472 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 545) took 1816 ms. Trace:
I20251024 19:48:10.586572 1774 rpcz_store.cc:276] 1024 19:48:08.769900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.825908 (+ 56008us) service_pool.cc:225] Handling call
1024 19:48:10.586346 (+1760438us) ts_tablet_manager.cc:1151] Deleting tablet 4c54e569156042fca5e9ab0feaedf632
1024 19:48:10.586359 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.586452 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1749139}
W20251024 19:48:10.587041 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4c54e569156042fca5e9ab0feaedf632 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c54e569156042fca5e9ab0feaedf632 already in progress: creating tablet
I20251024 19:48:10.587399 1774 tablet_replica.cc:333] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:10.587639 1774 raft_consensus.cc:2243] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:10.587819 1774 pending_rounds.cc:70] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:10.587872 1774 pending_rounds.cc:77] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429343402405888 op_type: NO_OP noop_request { }
I20251024 19:48:10.587991 1774 raft_consensus.cc:2889] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:10.588097 1774 raft_consensus.cc:2272] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.588285 1768 tablet_bootstrap.cc:654] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.591871 1774 ts_tablet_manager.cc:1916] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.593108 1480 tablet_service.cc:1505] Processing CreateTablet for tablet e602f14c45104517b7618b163519a499 (DEFAULT_TABLE table=default.Table_d10102cfa0fa4eb9b806bacdd5066807 [id=de0ca8d2f97b4354801830c6b614db9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.595551 1489 tablet_service.cc:1505] Processing CreateTablet for tablet bf241a9566c34b7b911b69aaf4de06a2 (DEFAULT_TABLE table=default.Table_3abf70710bce4728bfecd4a89d31dea6 [id=19704cecc9eb4063a579e16c491641df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.596072 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4ef1ab2b2d2c41f09c96c895942ef743
I20251024 19:48:10.598912 1189 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet ce0ef4443e574846a9f48fef3b893352 (table default.Table_f203b363de4c47fca0a0599140d18279 [id=ec57efb4d37e44f1b68045f58bc11c45]). Expected version 1 got 0
I20251024 19:48:10.599171 1189 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet b20a6845ae1645c0b51f096d471d4ccc (table default.Table_50e12cf5f39d4fa2983835ef0dbbc3d6 [id=201e5232763a4940a1288a56a695f71c]). Expected version 1 got 0
I20251024 19:48:10.599354 1189 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 037c8cc1e53447109fafc047c02f196f (table default.Table_e87be853efba49aeb50e71270beac600 [id=5bc052b3e0f04699b4b391bcec33617d]). Expected version 1 got 0
W20251024 19:48:10.600301 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa9c5b8254d3427f907879af997ac112 consensus state has no leader
I20251024 19:48:10.600616 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet 57f522a825264ea7b215304dbb7aa2d5 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:10.600934 1766 ts_tablet_manager.cc:1403] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.569s user 0.001s sys 0.010s
I20251024 19:48:10.601344 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ec9fec2116ab4e40bff85f19b11d6f73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:10.602018 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec9fec2116ab4e40bff85f19b11d6f73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:10.604907 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bacc8601ca1437586bdc9a52e086299. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:10.605924 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet b22b58404b8b441ba97ef536885a66e0 (table default.Table_f16963cab00043a4887d93a6326a1d9c [id=bdfe9c0f34ac41eebb843bd17fa9635c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b22b58404b8b441ba97ef536885a66e0 consensus state has no leader
I20251024 19:48:10.606184 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ef1ab2b2d2c41f09c96c895942ef743 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:10.608304 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e602f14c45104517b7618b163519a499. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.608934 2584 raft_consensus.cc:493] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:10.609133 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7255804242d424f8cbf89ef57d4ff9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.609084 2584 raft_consensus.cc:515] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.609812 2584 leader_election.cc:290] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:10.609910 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf241a9566c34b7b911b69aaf4de06a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.611555 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ef1ab2b2d2c41f09c96c895942ef743 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:10.612560 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e1e21beee44847b1ebeb746f02c372. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.615343 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_dcb0f784408346bda089b634ed26412e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:10.616142 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcb0f784408346bda089b634ed26412e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:10.619720 1472 tablet_service.cc:1505] Processing CreateTablet for tablet e51e597b8fd044e393a3af07da25586c (DEFAULT_TABLE table=default.Table_1dbcae61809349c59e959bff6c5cdf24 [id=64c3c3285f5a4c36bc31c5577e3695cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.507265 1586 leader_election.cc:304] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
W20251024 19:48:10.621336 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 114106us
I20251024 19:48:10.621635 1586 leader_election.cc:304] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:10.622373 2501 raft_consensus.cc:2804] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:10.623454 2584 raft_consensus.cc:2804] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:10.623836 2501 raft_consensus.cc:697] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:10.624226 2501 consensus_queue.cc:237] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.585031 1776 ts_tablet_manager.cc:1929] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:10.633188 1768 tablet_bootstrap.cc:492] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:10.633419 1768 ts_tablet_manager.cc:1403] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.088s user 0.003s sys 0.000s
I20251024 19:48:10.634377 1768 raft_consensus.cc:359] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.634691 1768 raft_consensus.cc:385] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.634799 1768 raft_consensus.cc:740] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.635097 1768 consensus_queue.cc:260] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.635720 1768 ts_tablet_manager.cc:1434] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:10.636010 1768 tablet_bootstrap.cc:492] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:10.637497 1768 tablet_bootstrap.cc:654] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.632506 1776 log.cc:1199] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/73ffe342f541447d8519ee24bc98a0f3
I20251024 19:48:10.638039 1776 ts_tablet_manager.cc:1950] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:10.639503 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 624) took 1556 ms. Trace:
I20251024 19:48:10.639691 1776 rpcz_store.cc:276] 1024 19:48:09.082649 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.083590 (+ 941us) service_pool.cc:225] Handling call
1024 19:48:10.522689 (+1439099us) ts_tablet_manager.cc:1151] Deleting tablet 73ffe342f541447d8519ee24bc98a0f3
1024 19:48:10.522703 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.581336 (+ 58633us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.584919 (+ 3583us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.639481 (+ 54562us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438770}
I20251024 19:48:10.640187 1776 tablet_replica.cc:333] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:10.640513 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 73ffe342f541447d8519ee24bc98a0f3 (table default.Table_74e4f24f12db403686548924c83c5191 [id=508d6cdc5d4a494bad31eaf36383e214]) successfully deleted
I20251024 19:48:10.638468 1586 leader_election.cc:304] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:10.645761 1776 raft_consensus.cc:2243] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:10.645931 1776 raft_consensus.cc:2272] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.646090 2638 raft_consensus.cc:2804] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:10.646368 1586 leader_election.cc:304] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:10.647495 1768 tablet_bootstrap.cc:492] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:10.647636 1768 ts_tablet_manager.cc:1403] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20251024 19:48:10.648384 1768 raft_consensus.cc:359] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:10.648552 1768 raft_consensus.cc:385] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.648603 1768 raft_consensus.cc:740] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.646517 1776 ts_tablet_manager.cc:1916] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.646646 2638 raft_consensus.cc:697] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:10.649010 2638 consensus_queue.cc:237] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.649700 1768 consensus_queue.cc:260] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:10.646812 1586 raft_consensus.cc:2686] T 73ffe342f541447d8519ee24bc98a0f3 P af3952de050140d383f6b0de7666fa97: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251024 19:48:10.654959 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c7dbe9ff2c4869994c4f43d5676c30" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:10.657809 1586 leader_election.cc:343] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.658950 1768 ts_tablet_manager.cc:1434] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251024 19:48:10.662883 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d18d308917fc44cbb7f0af85912d01c8" } new_table_name: "default.Table_dde32a1ec1eb47359dc5975ed58e9faa" modify_external_catalogs: true
I20251024 19:48:10.620129 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e51e597b8fd044e393a3af07da25586c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.665053 1768 tablet_bootstrap.cc:492] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:10.620610 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 7cab9c17496947efbbe3cd9e5d8a49a8 (DEFAULT_TABLE table=default.Table_fc2a1aeddc444f578693d922323862f5 [id=8a834f513c0045ca874a0d76e08d494c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.620730 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d39eb2f825b417a9361c6f4876e0cd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.666054 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet c0f231f739b64f9fb6a726589a213146 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.666183 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet fd52295b14b14e70bbdcb45f1bd849e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.666290 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 7be9290daabb404f955f05922d897fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.666388 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e0dc7d4fed5448cb616af29e405e766 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.666709 1486 tablet_service.cc:1505] Processing CreateTablet for tablet c0f0c6c4a3f04b079a2a2c0b1b2050b7 (DEFAULT_TABLE table=default.Table_86395810be7844a68994a4a5bbf0ab32 [id=b5a5fe39570b4001b750c392d2823d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.667306 1768 tablet_bootstrap.cc:654] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.669013 1774 ts_tablet_manager.cc:1929] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:10.669117 1774 log.cc:1199] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/5368789df6024d7daa9ccc2257137d58
I20251024 19:48:10.669498 1774 ts_tablet_manager.cc:1950] T 5368789df6024d7daa9ccc2257137d58 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:10.656112 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c7dbe9ff2c4869994c4f43d5676c30" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:10.670245 1508 raft_consensus.cc:2468] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:10.670646 1586 leader_election.cc:304] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:10.670992 2569 raft_consensus.cc:2804] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:10.671082 2569 raft_consensus.cc:493] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:10.671156 2569 raft_consensus.cc:3060] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:10.674744 1776 ts_tablet_manager.cc:1929] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:10.674853 1776 log.cc:1199] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/ad81d80697984ff1834a7d795e0f4b9b
I20251024 19:48:10.675181 1776 ts_tablet_manager.cc:1950] T ad81d80697984ff1834a7d795e0f4b9b P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:10.675879 2569 raft_consensus.cc:515] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.676687 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 550) took 1749 ms. Trace:
I20251024 19:48:10.676787 1774 rpcz_store.cc:276] 1024 19:48:08.927255 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.927314 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:10.587341 (+1660027us) ts_tablet_manager.cc:1151] Deleting tablet 5368789df6024d7daa9ccc2257137d58
1024 19:48:10.587353 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.626281 (+ 38928us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.668912 (+ 42631us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.676663 (+ 7751us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1659798}
I20251024 19:48:10.677266 2569 leader_election.cc:290] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:10.677255 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 5368789df6024d7daa9ccc2257137d58 (table default.Table_a81e7aae0a7a4f3db5c84efc4b16f873 [id=16bcec63f27d4dcd826ef11d45d58400]) successfully deleted
I20251024 19:48:10.677824 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c7dbe9ff2c4869994c4f43d5676c30" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:10.678308 1586 leader_election.cc:343] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.678399 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c7dbe9ff2c4869994c4f43d5676c30" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:10.678433 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 551) took 1722 ms. Trace:
I20251024 19:48:10.678620 1508 raft_consensus.cc:3060] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:10.678530 1774 rpcz_store.cc:276] 1024 19:48:08.955813 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.955879 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:10.678323 (+1722444us) ts_tablet_manager.cc:1151] Deleting tablet 23d4dd0b42d343cb9354ac698a8c761f
1024 19:48:10.678333 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.678418 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1722200}
W20251024 19:48:10.679021 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 23d4dd0b42d343cb9354ac698a8c761f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23d4dd0b42d343cb9354ac698a8c761f already in progress: creating tablet
I20251024 19:48:10.679374 1774 tablet_replica.cc:333] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:10.679625 1774 raft_consensus.cc:2243] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:10.679772 1774 raft_consensus.cc:2272] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.680578 1774 ts_tablet_manager.cc:1916] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.682271 1508 raft_consensus.cc:2468] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:10.682708 1586 leader_election.cc:304] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:10.683138 2569 raft_consensus.cc:2804] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:10.683293 2569 raft_consensus.cc:697] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:10.683578 2569 consensus_queue.cc:237] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.686836 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 625) took 1602 ms. Trace:
I20251024 19:48:10.686960 1776 rpcz_store.cc:276] 1024 19:48:09.084776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.085049 (+ 273us) service_pool.cc:225] Handling call
1024 19:48:10.640133 (+1555084us) ts_tablet_manager.cc:1151] Deleting tablet ad81d80697984ff1834a7d795e0f4b9b
1024 19:48:10.640144 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.671236 (+ 31092us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.674657 (+ 3421us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.686809 (+ 12152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1554819}
I20251024 19:48:10.687470 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet ad81d80697984ff1834a7d795e0f4b9b (table default.Table_61b7f6c428aa4bd9a65c1d66171c59c9 [id=6c604c6f86f944a285866376d9a5ce65]) successfully deleted
W20251024 19:48:10.687794 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 59956dad06064024ab171631ef6c1dca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59956dad06064024ab171631ef6c1dca already in progress: creating tablet
W20251024 19:48:10.688149 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d2c92975cf66441f9ac90c19772d3db3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2c92975cf66441f9ac90c19772d3db3 already in progress: creating tablet
I20251024 19:48:10.688509 1776 tablet_replica.cc:333] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:48:10.688433 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2087e295ad4b4daa8eeebd49b43e08c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2087e295ad4b4daa8eeebd49b43e08c8 already in progress: creating tablet
I20251024 19:48:10.688679 1776 raft_consensus.cc:2243] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:10.688824 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 6f89a3bbeb9b4d57a49078729110d951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f89a3bbeb9b4d57a49078729110d951 already in progress: creating tablet
W20251024 19:48:10.689095 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 64499c8734d34c66b0b52a2cd3ef0673 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64499c8734d34c66b0b52a2cd3ef0673 already in progress: creating tablet
W20251024 19:48:10.689458 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2d39eb2f825b417a9361c6f4876e0cd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d39eb2f825b417a9361c6f4876e0cd7 already in progress: creating tablet
W20251024 19:48:10.689812 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet c0f231f739b64f9fb6a726589a213146 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0f231f739b64f9fb6a726589a213146 already in progress: creating tablet
I20251024 19:48:10.690397 1766 raft_consensus.cc:359] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.690579 1766 raft_consensus.cc:385] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.690630 1766 raft_consensus.cc:740] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.690852 1473 tablet_service.cc:1505] Processing CreateTablet for tablet de2ebb79d4fe4a1bbac372bb26f370ed (DEFAULT_TABLE table=default.Table_b9cb3ccac55f4b0a8897f5cd1412440d [id=c329ff0d97454526835cdda8938deeb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.690825 1766 consensus_queue.cc:260] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.691188 2613 raft_consensus.cc:493] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:10.691327 1766 ts_tablet_manager.cc:1434] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.090s user 0.000s sys 0.001s
I20251024 19:48:10.691295 2613 raft_consensus.cc:515] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.688833 1776 raft_consensus.cc:2272] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.691895 2613 leader_election.cc:290] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:10.692997 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc14b0cdb7f342ac956a7857ab3f35d7" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:10.693728 1774 ts_tablet_manager.cc:1929] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:10.693841 1774 log.cc:1199] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/9fffc42ddda14bd28800917ff2f0ced5
W20251024 19:48:10.693821 1447 leader_election.cc:343] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.694164 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de2ebb79d4fe4a1bbac372bb26f370ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.694207 1774 ts_tablet_manager.cc:1950] T 9fffc42ddda14bd28800917ff2f0ced5 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:10.695068 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 552) took 1717 ms. Trace:
I20251024 19:48:10.695158 1774 rpcz_store.cc:276] 1024 19:48:08.977999 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.978061 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:10.679320 (+1701259us) ts_tablet_manager.cc:1151] Deleting tablet 9fffc42ddda14bd28800917ff2f0ced5
1024 19:48:10.679332 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.683793 (+ 4461us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.693600 (+ 9807us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.695045 (+ 1445us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1701001}
I20251024 19:48:10.695659 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 9fffc42ddda14bd28800917ff2f0ced5 (table default.Table_64aaab6eda0a443fbb49223835af43e3 [id=52efb5f413d14db28314ecbb058a351f]) successfully deleted
I20251024 19:48:10.699522 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cab9c17496947efbbe3cd9e5d8a49a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.699925 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 556) took 1652 ms. Trace:
I20251024 19:48:10.700014 1774 rpcz_store.cc:276] 1024 19:48:09.047219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.047530 (+ 311us) service_pool.cc:225] Handling call
1024 19:48:10.699776 (+1652246us) ts_tablet_manager.cc:1151] Deleting tablet 2bb5f84c1f104b338be71cb55ae77929
1024 19:48:10.699790 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.699903 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1596792}
W20251024 19:48:10.700471 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2bb5f84c1f104b338be71cb55ae77929 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bb5f84c1f104b338be71cb55ae77929 already in progress: creating tablet
I20251024 19:48:10.701413 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 557) took 1654 ms. Trace:
I20251024 19:48:10.701494 1774 rpcz_store.cc:276] 1024 19:48:09.047337 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.048033 (+ 696us) service_pool.cc:225] Handling call
1024 19:48:10.701235 (+1653202us) ts_tablet_manager.cc:1151] Deleting tablet e68997931ede4b6398570a7d08edaf4d
1024 19:48:10.701247 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.701391 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598182}
W20251024 19:48:10.702309 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet e68997931ede4b6398570a7d08edaf4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e68997931ede4b6398570a7d08edaf4d already in progress: creating tablet
I20251024 19:48:10.703191 1768 tablet_bootstrap.cc:492] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:10.703329 1768 ts_tablet_manager.cc:1403] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.004s
I20251024 19:48:10.704241 1768 raft_consensus.cc:359] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.704391 1768 raft_consensus.cc:385] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.704432 1768 raft_consensus.cc:740] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.704645 1768 consensus_queue.cc:260] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.708066 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f0c6c4a3f04b079a2a2c0b1b2050b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.710287 1768 ts_tablet_manager.cc:1434] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251024 19:48:10.710552 1775 ts_tablet_manager.cc:1929] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:10.710649 1775 log.cc:1199] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/44ee03ede02c4bdab706fde910978332
I20251024 19:48:10.710994 1775 ts_tablet_manager.cc:1950] T 44ee03ede02c4bdab706fde910978332 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:10.711772 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 483) took 4718 ms. Trace:
I20251024 19:48:10.717415 1768 tablet_bootstrap.cc:492] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:10.721844 1768 tablet_bootstrap.cc:654] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.722653 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 558) took 1675 ms. Trace:
I20251024 19:48:10.722754 1774 rpcz_store.cc:276] 1024 19:48:09.047466 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.048215 (+ 749us) service_pool.cc:225] Handling call
1024 19:48:10.722435 (+1674220us) ts_tablet_manager.cc:1151] Deleting tablet 95ae017a2e084ccabfa69c9a5befe0e7
1024 19:48:10.722499 (+ 64us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.722631 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1619336}
W20251024 19:48:10.723284 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 95ae017a2e084ccabfa69c9a5befe0e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95ae017a2e084ccabfa69c9a5befe0e7 already in progress: creating tablet
I20251024 19:48:10.723325 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 559) took 1650 ms. Trace:
I20251024 19:48:10.723404 1774 rpcz_store.cc:276] 1024 19:48:09.072945 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.072999 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:10.723201 (+1650202us) ts_tablet_manager.cc:1151] Deleting tablet 1e6a588092a44688bc96f16ac00e39f9
1024 19:48:10.723213 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.723308 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1620058}
W20251024 19:48:10.723881 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 1e6a588092a44688bc96f16ac00e39f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e6a588092a44688bc96f16ac00e39f9 already in progress: creating tablet
I20251024 19:48:10.725034 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 585) took 1077 ms. Trace:
I20251024 19:48:10.725111 1774 rpcz_store.cc:276] 1024 19:48:09.647562 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.647636 (+ 74us) service_pool.cc:225] Handling call
1024 19:48:10.724830 (+1077194us) ts_tablet_manager.cc:1151] Deleting tablet d2c92975cf66441f9ac90c19772d3db3
1024 19:48:10.724839 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.724909 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1076962}
I20251024 19:48:10.725142 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc14b0cdb7f342ac956a7857ab3f35d7" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:10.725494 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet d2c92975cf66441f9ac90c19772d3db3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2c92975cf66441f9ac90c19772d3db3 already in progress: creating tablet
I20251024 19:48:10.726250 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 584) took 1078 ms. Trace:
I20251024 19:48:10.726336 1774 rpcz_store.cc:276] 1024 19:48:09.647393 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.647481 (+ 88us) service_pool.cc:225] Handling call
1024 19:48:10.726128 (+1078647us) ts_tablet_manager.cc:1151] Deleting tablet 59956dad06064024ab171631ef6c1dca
1024 19:48:10.726138 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.726228 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1076397}
W20251024 19:48:10.726802 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 59956dad06064024ab171631ef6c1dca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59956dad06064024ab171631ef6c1dca already in progress: creating tablet
I20251024 19:48:10.729424 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 587) took 1079 ms. Trace:
I20251024 19:48:10.729530 1774 rpcz_store.cc:276] 1024 19:48:09.649838 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.649903 (+ 65us) service_pool.cc:225] Handling call
1024 19:48:10.729326 (+1079423us) ts_tablet_manager.cc:1151] Deleting tablet 2087e295ad4b4daa8eeebd49b43e08c8
1024 19:48:10.729334 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.729407 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1079259}
I20251024 19:48:10.729947 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 588) took 1080 ms. Trace:
I20251024 19:48:10.730015 1774 rpcz_store.cc:276] 1024 19:48:09.650004 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.650098 (+ 94us) service_pool.cc:225] Handling call
1024 19:48:10.729864 (+1079766us) ts_tablet_manager.cc:1151] Deleting tablet 6f89a3bbeb9b4d57a49078729110d951
1024 19:48:10.729872 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.729935 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1079637}
I20251024 19:48:10.730494 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 586) took 1080 ms. Trace:
I20251024 19:48:10.730573 1774 rpcz_store.cc:276] 1024 19:48:09.649681 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.650222 (+ 541us) service_pool.cc:225] Handling call
1024 19:48:10.730365 (+1080143us) ts_tablet_manager.cc:1151] Deleting tablet 64499c8734d34c66b0b52a2cd3ef0673
1024 19:48:10.730376 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.730476 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1079950}
I20251024 19:48:10.731273 1774 tablet_replica.cc:333] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:10.731523 1774 raft_consensus.cc:2243] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:10.731706 1774 raft_consensus.cc:2272] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.732367 1774 ts_tablet_manager.cc:1916] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:10.733834 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2087e295ad4b4daa8eeebd49b43e08c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2087e295ad4b4daa8eeebd49b43e08c8 already in progress: creating tablet
W20251024 19:48:10.734158 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 6f89a3bbeb9b4d57a49078729110d951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f89a3bbeb9b4d57a49078729110d951 already in progress: creating tablet
W20251024 19:48:10.734738 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 64499c8734d34c66b0b52a2cd3ef0673 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64499c8734d34c66b0b52a2cd3ef0673 already in progress: creating tablet
W20251024 19:48:10.734972 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2d39eb2f825b417a9361c6f4876e0cd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d39eb2f825b417a9361c6f4876e0cd7 already in progress: creating tablet
I20251024 19:48:10.737479 2546 raft_consensus.cc:493] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:10.737666 2546 raft_consensus.cc:515] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.738682 2546 leader_election.cc:290] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:10.714030 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 839cef50795840fab564b94d88565f8c (DEFAULT_TABLE table=default.Table_e97c30902b564ba2a411ea987cb0ef70 [id=e95322d516ba4098a327e991c8834bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:10.740994 1775 rpcz_store.cc:276] 1024 19:48:05.993073 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.993760 (+ 687us) service_pool.cc:225] Handling call
1024 19:48:10.113101 (+4119341us) ts_tablet_manager.cc:1151] Deleting tablet 44ee03ede02c4bdab706fde910978332
1024 19:48:10.113113 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.554851 (+441738us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.710438 (+155587us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.711752 (+ 1314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4119147}
W20251024 19:48:10.741094 1305 leader_election.cc:343] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: fca73ab61f0c47df9186459caf2b9a32
I20251024 19:48:10.741261 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 839cef50795840fab564b94d88565f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.741339 1775 tablet_replica.cc:333] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:10.741518 1775 raft_consensus.cc:2243] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:10.741874 1775 raft_consensus.cc:2272] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.742409 1775 ts_tablet_manager.cc:1916] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.747035 1768 tablet_bootstrap.cc:492] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:10.747190 1768 ts_tablet_manager.cc:1403] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.001s
I20251024 19:48:10.719919 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 069091f1c238487580baab38af6a70e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.720221 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet ae0d26f010bd46d3a05ac50f966d5769 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.720768 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 55aa786b9f6847baa265802cc28b2b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.721556 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 27f1a5eccdf6430c9e220988043596b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.722220 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ef1ab2b2d2c41f09c96c895942ef743 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:10 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:10.723800 1766 tablet_bootstrap.cc:492] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:10.748174 1768 raft_consensus.cc:359] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.748364 1768 raft_consensus.cc:385] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.748420 1768 raft_consensus.cc:740] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.748661 1768 consensus_queue.cc:260] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.749706 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 44ee03ede02c4bdab706fde910978332 (table default.Table_5f1d0b12b2034f5185f21c87157cf051 [id=46806f1d4f434926bc23214f0e3605d8]) successfully deleted
W20251024 19:48:10.725668 1444 leader_election.cc:343] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.750578 1444 leader_election.cc:304] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:10.751693 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fca73ab61f0c47df9186459caf2b9a32" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:10.752038 1510 raft_consensus.cc:2393] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
I20251024 19:48:10.752770 2613 raft_consensus.cc:2749] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:10.753611 1766 tablet_bootstrap.cc:654] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.754097 1308 leader_election.cc:304] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:10.754595 2546 raft_consensus.cc:3060] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:10.755554 1768 ts_tablet_manager.cc:1434] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251024 19:48:10.760668 2546 raft_consensus.cc:2749] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:10.761844 1768 tablet_bootstrap.cc:492] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:10.764138 1768 tablet_bootstrap.cc:654] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.765923 1776 ts_tablet_manager.cc:1916] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.770159 2546 raft_consensus.cc:493] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251024 19:48:10.770408 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 63675.8us
I20251024 19:48:10.770340 2546 raft_consensus.cc:515] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.772387 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b20a6845ae1645c0b51f096d471d4ccc" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:10.773830 2546 leader_election.cc:290] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:10.774219 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b20a6845ae1645c0b51f096d471d4ccc" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:10.774821 1305 leader_election.cc:304] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:10.775348 1776 ts_tablet_manager.cc:1929] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:10.775375 2546 raft_consensus.cc:2749] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:10.776865 1369 raft_consensus.cc:3060] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:10.777410 1776 log.cc:1199] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/fd52295b14b14e70bbdcb45f1bd849e4
I20251024 19:48:10.777904 1776 ts_tablet_manager.cc:1950] T fd52295b14b14e70bbdcb45f1bd849e4 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:10.779114 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet fd52295b14b14e70bbdcb45f1bd849e4 (table default.Table_ff7a43292b9045f9bbfa71883bcd7e61 [id=e2c539db2aa148448925a69f4848feb2]) successfully deleted
I20251024 19:48:10.779209 1775 ts_tablet_manager.cc:1929] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:10.779242 1774 ts_tablet_manager.cc:1929] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:10.779302 1775 log.cc:1199] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/dc1b6f0a377945089c50012f74d38564
I20251024 19:48:10.779330 1774 log.cc:1199] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/fd52295b14b14e70bbdcb45f1bd849e4
I20251024 19:48:10.779637 1775 ts_tablet_manager.cc:1950] T dc1b6f0a377945089c50012f74d38564 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:10.779733 1774 ts_tablet_manager.cc:1950] T fd52295b14b14e70bbdcb45f1bd849e4 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:10.780391 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 484) took 4787 ms. Trace:
I20251024 19:48:10.780820 2584 raft_consensus.cc:697] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:10.782063 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet fd52295b14b14e70bbdcb45f1bd849e4 (table default.Table_ff7a43292b9045f9bbfa71883bcd7e61 [id=e2c539db2aa148448925a69f4848feb2]) successfully deleted
W20251024 19:48:10.782255 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7be9290daabb404f955f05922d897fbf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7be9290daabb404f955f05922d897fbf already in progress: creating tablet
W20251024 19:48:10.782336 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet c0f231f739b64f9fb6a726589a213146 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0f231f739b64f9fb6a726589a213146 already in progress: creating tablet
W20251024 19:48:10.782580 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7e0dc7d4fed5448cb616af29e405e766 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e0dc7d4fed5448cb616af29e405e766 already in progress: creating tablet
W20251024 19:48:10.782848 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 069091f1c238487580baab38af6a70e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 069091f1c238487580baab38af6a70e2 already in progress: creating tablet
I20251024 19:48:10.783083 1774 tablet_replica.cc:333] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:10.783281 1774 raft_consensus.cc:2243] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:10.783269 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet ae0d26f010bd46d3a05ac50f966d5769 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae0d26f010bd46d3a05ac50f966d5769 already in progress: creating tablet
I20251024 19:48:10.783428 1774 raft_consensus.cc:2272] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.784071 1774 ts_tablet_manager.cc:1916] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.782979 2584 consensus_queue.cc:237] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:10.787401 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 55aa786b9f6847baa265802cc28b2b80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55aa786b9f6847baa265802cc28b2b80 already in progress: creating tablet
W20251024 19:48:10.788218 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 27f1a5eccdf6430c9e220988043596b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27f1a5eccdf6430c9e220988043596b0 already in progress: creating tablet
W20251024 19:48:10.788483 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4ef1ab2b2d2c41f09c96c895942ef743 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ef1ab2b2d2c41f09c96c895942ef743 already in progress: creating tablet
I20251024 19:48:10.789626 1768 tablet_bootstrap.cc:492] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:10.789783 1768 ts_tablet_manager.cc:1403] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20251024 19:48:10.791376 1768 raft_consensus.cc:359] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.791833 1768 raft_consensus.cc:385] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.791914 1768 raft_consensus.cc:740] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.792151 1768 consensus_queue.cc:260] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.792768 1768 ts_tablet_manager.cc:1434] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251024 19:48:10.793443 1768 tablet_bootstrap.cc:492] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:10.795974 1768 tablet_bootstrap.cc:654] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.802521 1767 tablet_bootstrap.cc:492] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:10.802745 1767 ts_tablet_manager.cc:1403] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.416s user 0.028s sys 0.019s
I20251024 19:48:10.803594 1767 raft_consensus.cc:359] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.809065 1767 raft_consensus.cc:385] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.809234 1767 raft_consensus.cc:740] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.809906 2546 raft_consensus.cc:493] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:10.810032 2546 raft_consensus.cc:515] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:10.811411 1308 leader_election.cc:343] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 6ec5c7bf36874423b4790c6169da8bd0
I20251024 19:48:10.812264 2546 leader_election.cc:290] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:10.812857 1305 leader_election.cc:343] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 6ec5c7bf36874423b4790c6169da8bd0
I20251024 19:48:10.812955 1305 leader_election.cc:304] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:10.813239 1774 ts_tablet_manager.cc:1929] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:10.813361 2546 raft_consensus.cc:2749] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:10.813374 1774 log.cc:1199] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/fca73ab61f0c47df9186459caf2b9a32
I20251024 19:48:10.809648 1767 consensus_queue.cc:260] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.816562 1774 ts_tablet_manager.cc:1950] T fca73ab61f0c47df9186459caf2b9a32 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:10.816941 1767 ts_tablet_manager.cc:1434] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251024 19:48:10.817297 1767 tablet_bootstrap.cc:492] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:10.818578 1768 tablet_bootstrap.cc:492] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:10.818711 1768 ts_tablet_manager.cc:1403] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20251024 19:48:10.818356 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet fca73ab61f0c47df9186459caf2b9a32 (table default.Table_420ebc050cf440e9a59a18e26f7b83af [id=be1cc9d047554ea6a9afa531fb08a81d]) successfully deleted
W20251024 19:48:10.819502 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7be9290daabb404f955f05922d897fbf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7be9290daabb404f955f05922d897fbf already in progress: creating tablet
I20251024 19:48:10.819638 1768 raft_consensus.cc:359] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.819854 1768 raft_consensus.cc:385] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.819922 1768 raft_consensus.cc:740] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.820154 1768 consensus_queue.cc:260] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.820719 1768 ts_tablet_manager.cc:1434] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:10.821509 1768 tablet_bootstrap.cc:492] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:10.821564 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4846799df5b242ce8f19c034320c8a1d consensus state has no leader
I20251024 19:48:10.822010 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ebaca4c94c0a4718bce68ebbc4feb98a" } new_table_name: "default.Table_aa8adfc621f04c168f5925625cd55caa" modify_external_catalogs: true
I20251024 19:48:10.823235 1768 tablet_bootstrap.cc:654] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.823701 2546 raft_consensus.cc:493] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:10.823863 2546 raft_consensus.cc:515] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:10.825917 2546 leader_election.cc:290] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:10.829351 1305 leader_election.cc:343] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: ca0527af718f494a8bbcc66e7b325bed
W20251024 19:48:10.820104 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7e0dc7d4fed5448cb616af29e405e766 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e0dc7d4fed5448cb616af29e405e766 already in progress: creating tablet
W20251024 19:48:10.829918 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet ae0d26f010bd46d3a05ac50f966d5769 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae0d26f010bd46d3a05ac50f966d5769 already in progress: creating tablet
W20251024 19:48:10.830214 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 55aa786b9f6847baa265802cc28b2b80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55aa786b9f6847baa265802cc28b2b80 already in progress: creating tablet
W20251024 19:48:10.830457 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 069091f1c238487580baab38af6a70e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 069091f1c238487580baab38af6a70e2 already in progress: creating tablet
W20251024 19:48:10.830749 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 27f1a5eccdf6430c9e220988043596b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27f1a5eccdf6430c9e220988043596b0 already in progress: creating tablet
W20251024 19:48:10.830986 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4ef1ab2b2d2c41f09c96c895942ef743 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ef1ab2b2d2c41f09c96c895942ef743 already in progress: creating tablet
I20251024 19:48:10.836026 1767 tablet_bootstrap.cc:654] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.842164 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a5046911ec4642bd811676fbd8782d9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:10.845163 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5046911ec4642bd811676fbd8782d9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:10.850196 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3834cd40435b4874a48fbec099663c9b" } new_table_name: "default.Table_96f18cad8ca2451e9e8a9b82da4754eb" modify_external_catalogs: true
I20251024 19:48:10.780622 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a9b28ff5674d5db8002b748e2733b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:10.786847 1766 tablet_bootstrap.cc:492] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:10.853790 1766 ts_tablet_manager.cc:1403] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.130s user 0.002s sys 0.007s
I20251024 19:48:10.854813 1766 raft_consensus.cc:359] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.855042 1766 raft_consensus.cc:385] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.855096 1766 raft_consensus.cc:740] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.855346 1766 consensus_queue.cc:260] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.856012 1766 ts_tablet_manager.cc:1434] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251024 19:48:10.856374 1766 tablet_bootstrap.cc:492] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:10.825620 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca0527af718f494a8bbcc66e7b325bed" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:10.857780 1510 raft_consensus.cc:2393] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:10.858392 2610 tablet.cc:1722] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:10.859093 1308 leader_election.cc:304] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:10.864773 2613 raft_consensus.cc:493] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:10.864926 2613 raft_consensus.cc:515] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.865550 2613 leader_election.cc:290] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:10.865967 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "187925082cdd4923a6e9148bf9f270fd" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:10.865942 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_bb7717f8140d4bc6a90455b84c9906ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:10.866479 1444 leader_election.cc:343] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.870522 1766 tablet_bootstrap.cc:654] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.872927 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "187925082cdd4923a6e9148bf9f270fd" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:10.873363 2546 raft_consensus.cc:493] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:10.873513 2546 raft_consensus.cc:515] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:10.873615 1447 leader_election.cc:343] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.873539 2663 tablet.cc:1722] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:10.873749 1447 leader_election.cc:304] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:10.874140 2546 leader_election.cc:290] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:10.874302 2567 raft_consensus.cc:2749] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:10.875666 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d692e1988d13460fb1d54e930e17fc9d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:10.875967 1766 tablet_bootstrap.cc:492] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:10.876087 1766 ts_tablet_manager.cc:1403] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.005s
I20251024 19:48:10.876560 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d692e1988d13460fb1d54e930e17fc9d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:10.876889 1766 raft_consensus.cc:359] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.877051 1766 raft_consensus.cc:385] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.877108 1766 raft_consensus.cc:740] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.877162 1305 leader_election.cc:304] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:10.877296 1766 consensus_queue.cc:260] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.877771 2546 raft_consensus.cc:2749] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:10.877923 1766 ts_tablet_manager.cc:1434] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:10.878324 1766 tablet_bootstrap.cc:492] T 635a132735704cfda69bf7c8deaea81d P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:10.880395 1766 tablet_bootstrap.cc:654] T 635a132735704cfda69bf7c8deaea81d P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.881556 1768 tablet_bootstrap.cc:492] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:10.881810 1768 ts_tablet_manager.cc:1403] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.061s user 0.021s sys 0.018s
I20251024 19:48:10.882761 1768 raft_consensus.cc:359] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.883045 1768 raft_consensus.cc:385] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.883155 1768 raft_consensus.cc:740] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.883466 1768 consensus_queue.cc:260] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.884042 1768 ts_tablet_manager.cc:1434] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251024 19:48:10.884364 1768 tablet_bootstrap.cc:492] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:10.886817 1768 tablet_bootstrap.cc:654] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.892601 1767 tablet_bootstrap.cc:492] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:10.892943 1767 ts_tablet_manager.cc:1403] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.000s
I20251024 19:48:10.894316 1767 raft_consensus.cc:359] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.878470 1775 rpcz_store.cc:276] 1024 19:48:05.993173 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.993990 (+ 817us) service_pool.cc:225] Handling call
1024 19:48:10.741296 (+4747306us) ts_tablet_manager.cc:1151] Deleting tablet dc1b6f0a377945089c50012f74d38564
1024 19:48:10.741306 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:10.745412 (+ 4106us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.779113 (+ 33701us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.780369 (+ 1256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4747096}
I20251024 19:48:10.895233 1767 raft_consensus.cc:385] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.895578 1767 raft_consensus.cc:740] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.896278 1775 tablet_replica.cc:333] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:10.896342 1767 consensus_queue.cc:260] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.872932 2656 raft_consensus.cc:3060] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:10.900336 1775 raft_consensus.cc:2243] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:10.901053 1767 ts_tablet_manager.cc:1434] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251024 19:48:10.904369 1775 raft_consensus.cc:2272] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:10.906665 1775 ts_tablet_manager.cc:1916] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:10.916550 2656 raft_consensus.cc:2749] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:10.917837 1768 tablet_bootstrap.cc:492] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:10.918133 1768 ts_tablet_manager.cc:1403] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.004s
I20251024 19:48:10.919469 1768 raft_consensus.cc:359] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.919824 1768 raft_consensus.cc:385] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.919962 1768 raft_consensus.cc:740] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.920359 1768 consensus_queue.cc:260] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.921135 1768 ts_tablet_manager.cc:1434] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:10.933993 1767 tablet_bootstrap.cc:492] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:10.935503 1768 tablet_bootstrap.cc:492] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:10.942020 1767 tablet_bootstrap.cc:654] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.970654 1767 tablet_bootstrap.cc:492] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:10.907945 1766 tablet_bootstrap.cc:492] T 635a132735704cfda69bf7c8deaea81d P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:10.970974 1767 ts_tablet_manager.cc:1403] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.000s
I20251024 19:48:10.971079 1766 ts_tablet_manager.cc:1403] T 635a132735704cfda69bf7c8deaea81d P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.004s
I20251024 19:48:10.972568 1766 raft_consensus.cc:359] T 635a132735704cfda69bf7c8deaea81d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.972903 1766 raft_consensus.cc:385] T 635a132735704cfda69bf7c8deaea81d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.972977 1766 raft_consensus.cc:740] T 635a132735704cfda69bf7c8deaea81d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.972973 1767 raft_consensus.cc:359] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.973197 1767 raft_consensus.cc:385] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:10.973258 1767 raft_consensus.cc:740] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:10.951845 2613 raft_consensus.cc:493] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:10.973662 1767 consensus_queue.cc:260] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.973739 2613 raft_consensus.cc:515] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.974189 1767 ts_tablet_manager.cc:1434] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:10.974516 1767 tablet_bootstrap.cc:492] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:10.974637 2613 leader_election.cc:290] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:10.975781 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b79801594e95434cbb8fe7b75b14602e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:10.976668 1647 raft_consensus.cc:2468] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:10.976861 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b79801594e95434cbb8fe7b75b14602e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:10.977370 1444 leader_election.cc:304] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:10.978513 2613 raft_consensus.cc:2804] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:10.978615 2613 raft_consensus.cc:493] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:10.978691 2613 raft_consensus.cc:3060] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:10.979120 1447 leader_election.cc:343] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:10.973242 1766 consensus_queue.cc:260] T 635a132735704cfda69bf7c8deaea81d P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:10.982244 1766 ts_tablet_manager.cc:1434] T 635a132735704cfda69bf7c8deaea81d P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20251024 19:48:10.982630 1766 tablet_bootstrap.cc:492] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:10.985281 1766 tablet_bootstrap.cc:654] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.987115 1767 tablet_bootstrap.cc:654] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:10.966842 1488 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46462 (request call id 613) took 1612 ms. Trace:
I20251024 19:48:10.988998 1488 rpcz_store.cc:276] 1024 19:48:09.354412 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.356536 (+ 2124us) service_pool.cc:225] Handling call
1024 19:48:10.756461 (+1399925us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1024 19:48:10.756491 (+ 30us) ts_tablet_manager.cc:682] Creating new metadata...
1024 19:48:10.857872 (+101381us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:10.899839 (+ 41967us) tablet_replica.cc:206] Creating consensus instance
1024 19:48:10.966809 (+ 66970us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251024 19:48:10.995957 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb7717f8140d4bc6a90455b84c9906ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:10.998412 2613 raft_consensus.cc:515] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:10.999804 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet dc1b6f0a377945089c50012f74d38564 (table default.Table_52ec55c23c394294a68850ab6b33aabe [id=fb0e7cab20f44aaebdb09862bd193f41]) successfully deleted
W20251024 19:48:11.000069 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 104174us
I20251024 19:48:11.001948 2613 leader_election.cc:290] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:11.002763 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b79801594e95434cbb8fe7b75b14602e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:11.002959 1647 raft_consensus.cc:3060] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.006503 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b79801594e95434cbb8fe7b75b14602e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:11.007357 1447 leader_election.cc:343] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.008669 1767 tablet_bootstrap.cc:492] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:11.008797 1767 ts_tablet_manager.cc:1403] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.002s
I20251024 19:48:11.009565 1767 raft_consensus.cc:359] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.009748 1767 raft_consensus.cc:385] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.009799 1767 raft_consensus.cc:740] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.010030 1767 consensus_queue.cc:260] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.011781 1767 ts_tablet_manager.cc:1434] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:11.012858 1767 tablet_bootstrap.cc:492] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:11.014937 2663 tablet.cc:1722] T d692e1988d13460fb1d54e930e17fc9d P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:11.019948 2656 raft_consensus.cc:493] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.020205 2656 raft_consensus.cc:515] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.024106 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b96fc2309274048bed7a35e4bf4bae2" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:11.024402 1767 tablet_bootstrap.cc:654] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.023872 2675 tablet.cc:1722] T d692e1988d13460fb1d54e930e17fc9d P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251024 19:48:11.024655 1305 leader_election.cc:343] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 5b96fc2309274048bed7a35e4bf4bae2
I20251024 19:48:11.024819 1305 leader_election.cc:304] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:11.025111 2546 raft_consensus.cc:2749] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:11.025609 2656 leader_election.cc:290] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:11.026468 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e74966425ce8408da939910bc645012d" } modify_external_catalogs: true
I20251024 19:48:11.026742 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e74966425ce8408da939910bc645012d" } modify_external_catalogs: true
W20251024 19:48:11.029853 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14965e5ea7ae43f2b80986c58a2e9f2a consensus state has no leader
W20251024 19:48:11.030486 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7ba26769fe143ad9da73f9f4a7920cf consensus state has no leader
I20251024 19:48:11.030656 1184 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet ce0ef4443e574846a9f48fef3b893352 (table default.Table_f203b363de4c47fca0a0599140d18279 [id=ec57efb4d37e44f1b68045f58bc11c45]). Expected version 1 got 0
I20251024 19:48:11.031090 1184 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 187925082cdd4923a6e9148bf9f270fd (table default.Table_55f8c66bd97343438051a98af12d8c2c [id=33f5ac73298543d0b5af77a27e897b79]). Expected version 1 got 0
I20251024 19:48:11.032387 2638 raft_consensus.cc:493] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.032572 2638 raft_consensus.cc:515] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.033390 2638 leader_election.cc:290] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.033732 1187 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922] alter complete (version 1)
I20251024 19:48:10.970504 2667 tablet.cc:1722] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:11.035197 1623 tablet_service.cc:1505] Processing CreateTablet for tablet d391bc3b3a144415bd4e6b5fdcc37612 (DEFAULT_TABLE table=default.Table_7e1ad08d4bd3477494cf5688f7842dd0 [id=7c80bd1842d64d3db0984a72d4005d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.035631 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 2037fcca87b94b5fb31639b4d07307b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d129bdc876e425aabcf1c99997e319a) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.035856 1776 tablet_replica.cc:333] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:11.036024 1776 raft_consensus.cc:2243] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:11.036082 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 018d3e6aba6740f0bd82e8c83863bc4e (DEFAULT_TABLE table=default.Table_d343821349d34633bfa9da8bc9dad031 [id=d3a6e6ab22f3473e8353480218feecf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.036240 1776 pending_rounds.cc:70] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:11.036284 1776 pending_rounds.cc:77] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429351117078528 op_type: NO_OP noop_request { }
I20251024 19:48:11.036371 1776 raft_consensus.cc:2889] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:11.036458 1776 raft_consensus.cc:2272] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:11.036978 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdfa5eea3b674a3389a156d1c72204df" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:11.037369 1776 ts_tablet_manager.cc:1916] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:11.037519 1586 leader_election.cc:343] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.037752 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdfa5eea3b674a3389a156d1c72204df" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:11.038785 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 018d3e6aba6740f0bd82e8c83863bc4e (DEFAULT_TABLE table=default.Table_d343821349d34633bfa9da8bc9dad031 [id=d3a6e6ab22f3473e8353480218feecf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.039165 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 018d3e6aba6740f0bd82e8c83863bc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.040110 1766 tablet_bootstrap.cc:492] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:11.040241 1766 ts_tablet_manager.cc:1403] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20251024 19:48:11.041004 1766 raft_consensus.cc:359] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.041231 1766 raft_consensus.cc:385] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.041395 1766 raft_consensus.cc:740] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.041683 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fffc42ddda14bd28800917ff2f0ced5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.041934 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ec5c7bf36874423b4790c6169da8bd0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7712392f200842c4b728e019d6a3cfb0) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.042016 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bf08ccf7054fedb9efcbb00354c71a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.042162 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 5368789df6024d7daa9ccc2257137d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:58 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.042702 1774 tablet_replica.cc:333] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:11.042893 1774 raft_consensus.cc:2243] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:11.043033 1774 raft_consensus.cc:2272] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:11.043538 1774 ts_tablet_manager.cc:1916] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:11.045279 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet dc1b6f0a377945089c50012f74d38564 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ca8f818440f4ae7a63c0901fcdcdf59) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.038785 1349 tablet_service.cc:1505] Processing CreateTablet for tablet d391bc3b3a144415bd4e6b5fdcc37612 (DEFAULT_TABLE table=default.Table_7e1ad08d4bd3477494cf5688f7842dd0 [id=7c80bd1842d64d3db0984a72d4005d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.045636 1766 consensus_queue.cc:260] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.046000 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d391bc3b3a144415bd4e6b5fdcc37612. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.046186 1766 ts_tablet_manager.cc:1434] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20251024 19:48:11.046167 1179 catalog_manager.cc:4952] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9fffc42ddda14bd28800917ff2f0ced5 because the tablet was not found. No further retry: Not found: Tablet not found: 9fffc42ddda14bd28800917ff2f0ced5
I20251024 19:48:11.046479 1766 tablet_bootstrap.cc:492] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:11.047240 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 2cbeaac19e834ecf98e5b8afc952b9da (DEFAULT_TABLE table=default.Table_56bed2d1fdc64b03a33e778185a2f371 [id=dfe3b62a94814b0c93f82b17212ce65f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.047607 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cbeaac19e834ecf98e5b8afc952b9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.049949 1350 tablet_service.cc:1505] Processing CreateTablet for tablet cf18f375266b4448b3f67afd787df6ad (DEFAULT_TABLE table=default.Table_fe00154d4ddf4d90a36ab30b672cc5cf [id=8018e1cb82634a1da52c2a28a8b131ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.049922 1488 tablet_service.cc:1505] Processing CreateTablet for tablet d391bc3b3a144415bd4e6b5fdcc37612 (DEFAULT_TABLE table=default.Table_7e1ad08d4bd3477494cf5688f7842dd0 [id=7c80bd1842d64d3db0984a72d4005d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.050312 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf18f375266b4448b3f67afd787df6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.051123 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 4f00719c20934655bc744446194f7401 (DEFAULT_TABLE table=default.Table_fdc631c49a3d4e6b89b84a4af697b305 [id=f62531b6d396440e8f44ae943c4db403]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.051137 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 2cbeaac19e834ecf98e5b8afc952b9da (DEFAULT_TABLE table=default.Table_56bed2d1fdc64b03a33e778185a2f371 [id=dfe3b62a94814b0c93f82b17212ce65f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.051651 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d391bc3b3a144415bd4e6b5fdcc37612. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.051779 1616 tablet_service.cc:1505] Processing CreateTablet for tablet cf18f375266b4448b3f67afd787df6ad (DEFAULT_TABLE table=default.Table_fe00154d4ddf4d90a36ab30b672cc5cf [id=8018e1cb82634a1da52c2a28a8b131ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.054319 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 4f00719c20934655bc744446194f7401 (DEFAULT_TABLE table=default.Table_fdc631c49a3d4e6b89b84a4af697b305 [id=f62531b6d396440e8f44ae943c4db403]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.054668 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f00719c20934655bc744446194f7401. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.058434 1472 tablet_service.cc:1505] Processing CreateTablet for tablet 4f00719c20934655bc744446194f7401 (DEFAULT_TABLE table=default.Table_fdc631c49a3d4e6b89b84a4af697b305 [id=f62531b6d396440e8f44ae943c4db403]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.058434 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 018d3e6aba6740f0bd82e8c83863bc4e (DEFAULT_TABLE table=default.Table_d343821349d34633bfa9da8bc9dad031 [id=d3a6e6ab22f3473e8353480218feecf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.058948 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 2cbeaac19e834ecf98e5b8afc952b9da (DEFAULT_TABLE table=default.Table_56bed2d1fdc64b03a33e778185a2f371 [id=dfe3b62a94814b0c93f82b17212ce65f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.059441 1476 tablet_service.cc:1505] Processing CreateTablet for tablet cf18f375266b4448b3f67afd787df6ad (DEFAULT_TABLE table=default.Table_fe00154d4ddf4d90a36ab30b672cc5cf [id=8018e1cb82634a1da52c2a28a8b131ab]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:11.059516 1586 leader_election.cc:343] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.059628 1586 leader_election.cc:304] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:11.059863 2638 raft_consensus.cc:2749] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:11.060317 1767 tablet_bootstrap.cc:492] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:11.060532 1767 ts_tablet_manager.cc:1403] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.005s
I20251024 19:48:11.061658 1767 raft_consensus.cc:359] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.061887 1767 raft_consensus.cc:385] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.061970 1767 raft_consensus.cc:740] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.062223 1767 consensus_queue.cc:260] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.064733 1767 ts_tablet_manager.cc:1434] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251024 19:48:11.065392 1767 tablet_bootstrap.cc:492] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:11.066138 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fb1b7f0a2371441380ca5492cf1c6a3b" } modify_external_catalogs: true
I20251024 19:48:11.066380 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fb1b7f0a2371441380ca5492cf1c6a3b" } modify_external_catalogs: true
I20251024 19:48:11.066352 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9bfa6cbfde5148d78028b23509294e53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:11.066967 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bfa6cbfde5148d78028b23509294e53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.069445 1767 tablet_bootstrap.cc:654] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.070282 1766 tablet_bootstrap.cc:654] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.071812 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a83dd919e9ab42179bff96aac9a09cec" } modify_external_catalogs: true
I20251024 19:48:11.072021 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a83dd919e9ab42179bff96aac9a09cec" } modify_external_catalogs: true
I20251024 19:48:11.076512 1766 tablet_bootstrap.cc:492] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:11.076756 1766 ts_tablet_manager.cc:1403] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20251024 19:48:11.078512 1766 raft_consensus.cc:359] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.078895 1766 raft_consensus.cc:385] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.079010 1766 raft_consensus.cc:740] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.079362 1766 consensus_queue.cc:260] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.085559 1767 tablet_bootstrap.cc:492] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:11.085776 1767 ts_tablet_manager.cc:1403] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.002s
I20251024 19:48:11.086818 1767 raft_consensus.cc:359] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.087038 1767 raft_consensus.cc:385] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.087170 1767 raft_consensus.cc:740] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.087431 1767 consensus_queue.cc:260] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.087865 1766 ts_tablet_manager.cc:1434] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20251024 19:48:11.088783 1184 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet d692e1988d13460fb1d54e930e17fc9d (table default.Table_b09229e0e1254adeacfed952befaa65a [id=67a0be12c85648bdb4515a096db21d3b]). Expected version 1 got 0
I20251024 19:48:11.090116 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 018d3e6aba6740f0bd82e8c83863bc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.090586 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cbeaac19e834ecf98e5b8afc952b9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.092697 1774 ts_tablet_manager.cc:1929] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:11.092792 1774 log.cc:1199] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/6ec5c7bf36874423b4790c6169da8bd0
I20251024 19:48:11.093044 1774 ts_tablet_manager.cc:1950] T 6ec5c7bf36874423b4790c6169da8bd0 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:11.093292 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d391bc3b3a144415bd4e6b5fdcc37612. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.093721 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f00719c20934655bc744446194f7401. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.094127 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf18f375266b4448b3f67afd787df6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.097892 1647 raft_consensus.cc:2468] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:11.088337 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_466298cf05da40c1a6acf5e5fb65fa83" } modify_external_catalogs: true
I20251024 19:48:11.104212 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_466298cf05da40c1a6acf5e5fb65fa83" } modify_external_catalogs: true
I20251024 19:48:11.104360 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_082b885410b34b1cb2f154cdbd9700a8" } modify_external_catalogs: true
I20251024 19:48:11.105912 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_082b885410b34b1cb2f154cdbd9700a8" } modify_external_catalogs: true
I20251024 19:48:11.106455 1184 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet ce0ef4443e574846a9f48fef3b893352 (table default.Table_f203b363de4c47fca0a0599140d18279 [id=ec57efb4d37e44f1b68045f58bc11c45]). Expected version 1 got 0
I20251024 19:48:11.107141 1767 ts_tablet_manager.cc:1434] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20251024 19:48:11.107867 1767 tablet_bootstrap.cc:492] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:11.108511 1184 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 7503b22d9e12481b9af96385b9d76e50 (table default.Table_e5c532eb7fc241039c7e06e4757ea011 [id=9065c680f4da42b5a6e5ca335c8e2f3b]). Expected version 3 got 0
I20251024 19:48:11.109436 1184 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet a3259823b622461fa823841925b3d203 (table default.Table_ea0d3589c13a4ebeb950ddc737a9948b [id=350b37bf42194fcfbb660bffe83e47c4]). Expected version 4 got 0
I20251024 19:48:11.115582 2610 tablet.cc:1722] T ce0ef4443e574846a9f48fef3b893352 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:11.102469 1444 leader_election.cc:304] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:11.120541 2334 raft_consensus.cc:2804] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:11.120707 2334 raft_consensus.cc:697] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:11.120994 2334 consensus_queue.cc:237] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.123082 2656 raft_consensus.cc:493] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.123214 2656 raft_consensus.cc:515] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.123788 2656 leader_election.cc:290] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:11.114595 2675 tablet.cc:1722] T ce0ef4443e574846a9f48fef3b893352 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:11.125407 2613 raft_consensus.cc:493] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.125630 1768 tablet_bootstrap.cc:700] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532: Time spent opening tablet: real 0.188s user 0.002s sys 0.000s
I20251024 19:48:11.125540 2613 raft_consensus.cc:515] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.125689 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fd7726c532145c6b44ea9a37544510d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:11.126001 1768 tablet_bootstrap.cc:654] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.126137 2613 leader_election.cc:290] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.127709 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7352d84226db467dad31ae5941842b80" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:11.128614 1305 leader_election.cc:343] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 5fd7726c532145c6b44ea9a37544510d
I20251024 19:48:11.128736 1305 leader_election.cc:304] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:11.128947 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7352d84226db467dad31ae5941842b80" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:11.129124 1650 raft_consensus.cc:2468] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:11.129199 2656 raft_consensus.cc:2749] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:11.129496 1447 leader_election.cc:343] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.129703 1444 leader_election.cc:304] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:11.130157 2567 raft_consensus.cc:2804] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:11.130250 2567 raft_consensus.cc:493] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:11.130311 2567 raft_consensus.cc:3060] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.150360 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 6ec5c7bf36874423b4790c6169da8bd0 (table default.Table_52ecaf891a2449d78bb8a354e7fb904b [id=e0a69533d0fa41a4ad58f08e510fa39c]) successfully deleted
I20251024 19:48:11.150441 1767 tablet_bootstrap.cc:654] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:11.150862 1179 catalog_manager.cc:4952] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 5368789df6024d7daa9ccc2257137d58 because the tablet was not found. No further retry: Not found: Tablet not found: 5368789df6024d7daa9ccc2257137d58
I20251024 19:48:11.153833 1766 tablet_bootstrap.cc:492] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:11.157115 1766 tablet_bootstrap.cc:654] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.167503 1768 tablet_bootstrap.cc:492] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:11.167680 1768 ts_tablet_manager.cc:1403] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.232s user 0.006s sys 0.000s
I20251024 19:48:11.168380 1768 raft_consensus.cc:359] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.168540 1768 raft_consensus.cc:385] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.168589 1768 raft_consensus.cc:740] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.168900 1768 consensus_queue.cc:260] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.169363 1768 ts_tablet_manager.cc:1434] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:11.181553 2638 raft_consensus.cc:493] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.181761 2638 raft_consensus.cc:515] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.181010 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 28005301f9de48e5826f46c79aeb5fe5
W20251024 19:48:11.181198 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46e5e391a7674b5d8e1dd1a780fe36c8 consensus state has no leader
I20251024 19:48:11.182489 2638 leader_election.cc:290] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:11.182886 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3da81c18d9264f38966944ad13f256d8
I20251024 19:48:11.183456 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bf08ccf7054fedb9efcbb00354c71a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:11.183710 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet 3da81c18d9264f38966944ad13f256d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.183947 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bf08ccf7054fedb9efcbb00354c71a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:11.184105 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0dfdcd7f531e4ddca00916bf1dcec977" } new_table_name: "default.Table_b2510c82f11444d78f61be9b9f2fcaff" modify_external_catalogs: true
I20251024 19:48:11.184170 1510 raft_consensus.cc:2468] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
W20251024 19:48:11.184674 1586 leader_election.cc:343] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.184708 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 3da81c18d9264f38966944ad13f256d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.185189 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet 3da81c18d9264f38966944ad13f256d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:11.186901 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3da81c18d9264f38966944ad13f256d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3da81c18d9264f38966944ad13f256d8 already in progress: creating tablet
I20251024 19:48:11.186898 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e72407cfebf14ad2be5ee53e418fda62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:11.187595 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e72407cfebf14ad2be5ee53e418fda62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.190557 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 23a1b73314084848a4f28a042612f394
I20251024 19:48:11.181113 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 250f106664be4b5aaa330dbb4e43cb7b
I20251024 19:48:11.192631 1768 tablet_bootstrap.cc:492] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:11.193277 2663 tablet.cc:1722] T ce0ef4443e574846a9f48fef3b893352 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:11.195273 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet 46c1e2cef6cf4772b624060afb6fb117 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:11.197973 2567 raft_consensus.cc:515] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.198630 2567 leader_election.cc:290] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.198946 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cbeaac19e834ecf98e5b8afc952b9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.199431 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf18f375266b4448b3f67afd787df6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.200297 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7352d84226db467dad31ae5941842b80" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:11.200480 1650 raft_consensus.cc:3060] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.204867 1586 leader_election.cc:304] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:11.206491 2638 raft_consensus.cc:2804] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:11.206529 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet 250f106664be4b5aaa330dbb4e43cb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.206722 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 23a1b73314084848a4f28a042612f394 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.206574 2638 raft_consensus.cc:493] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:11.206924 2638 raft_consensus.cc:3060] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.207268 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 28005301f9de48e5826f46c79aeb5fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.207389 1767 tablet_bootstrap.cc:492] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:11.207958 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b9d913c84cec4d508db7fe4c3dc63e27" } new_table_name: "default.Table_693aa7a74dc241f9af879ec2c4cfd808" modify_external_catalogs: true
I20251024 19:48:11.208557 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ab36bd29826c4100968322ae6f68f6ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:11.209759 1767 ts_tablet_manager.cc:1403] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.102s user 0.007s sys 0.002s
I20251024 19:48:11.210029 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f00719c20934655bc744446194f7401. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.210582 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 018d3e6aba6740f0bd82e8c83863bc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.210726 1767 raft_consensus.cc:359] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.210886 1767 raft_consensus.cc:385] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.210938 1767 raft_consensus.cc:740] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.211148 1767 consensus_queue.cc:260] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.212498 1767 ts_tablet_manager.cc:1434] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251024 19:48:11.212814 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab36bd29826c4100968322ae6f68f6ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:11.213551 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d731403fbe0b4f92ac5899e13ea7bb19 consensus state has no leader
I20251024 19:48:11.214679 1766 tablet_bootstrap.cc:492] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:11.215408 1766 ts_tablet_manager.cc:1403] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.004s
W20251024 19:48:11.215557 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 46424.7us
I20251024 19:48:11.218003 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 250f106664be4b5aaa330dbb4e43cb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.218336 1766 raft_consensus.cc:359] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.218501 1766 raft_consensus.cc:385] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.218554 1766 raft_consensus.cc:740] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.218724 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet 250f106664be4b5aaa330dbb4e43cb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.218755 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7352d84226db467dad31ae5941842b80" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:11.218755 1766 consensus_queue.cc:260] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.219123 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet 23a1b73314084848a4f28a042612f394 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.220132 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7d5f945cec1f41f2abd299341bb4fff2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:11.220592 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 28005301f9de48e5826f46c79aeb5fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.220773 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 23a1b73314084848a4f28a042612f394 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:11.220755 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d5f945cec1f41f2abd299341bb4fff2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.220783 1183 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 635a132735704cfda69bf7c8deaea81d (table default.Table_933a6697624149369ab3ec28462b466a [id=93a61862707b4a7ba8a0ff349704dc2b]). Expected version 1 got 0
I20251024 19:48:11.221372 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet 28005301f9de48e5826f46c79aeb5fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.222260 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6d8df15af842462fb931f649dc0dd806"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:11.222788 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d8df15af842462fb931f649dc0dd806 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.221436 1183 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet ce0ef4443e574846a9f48fef3b893352 (table default.Table_f203b363de4c47fca0a0599140d18279 [id=ec57efb4d37e44f1b68045f58bc11c45]). Expected version 1 got 0
I20251024 19:48:11.224481 1766 ts_tablet_manager.cc:1434] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251024 19:48:11.224483 1183 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 037c8cc1e53447109fafc047c02f196f (table default.Table_e87be853efba49aeb50e71270beac600 [id=5bc052b3e0f04699b4b391bcec33617d]). Expected version 1 got 0
I20251024 19:48:11.225556 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 913534ab88dd459e9cac2faebb5492a1
I20251024 19:48:11.226622 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4be8c37e3871442480b6f9c25bb75797"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:11.227259 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4be8c37e3871442480b6f9c25bb75797 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:11.229398 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 250f106664be4b5aaa330dbb4e43cb7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 250f106664be4b5aaa330dbb4e43cb7b already in progress: creating tablet
I20251024 19:48:11.230088 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 913534ab88dd459e9cac2faebb5492a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.230444 1186 catalog_manager.cc:5649] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:11.230841 1186 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 7503b22d9e12481b9af96385b9d76e50 (table default.Table_e5c532eb7fc241039c7e06e4757ea011 [id=9065c680f4da42b5a6e5ca335c8e2f3b]). Expected version 3 got 0
I20251024 19:48:11.231083 1768 tablet_bootstrap.cc:654] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.231478 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet 913534ab88dd459e9cac2faebb5492a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.231709 1774 tablet_replica.cc:333] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:11.231758 1186 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 037c8cc1e53447109fafc047c02f196f (table default.Table_e87be853efba49aeb50e71270beac600 [id=5bc052b3e0f04699b4b391bcec33617d]). Expected version 1 got 0
I20251024 19:48:11.231913 1774 raft_consensus.cc:2243] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:11.232007 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 23a1b73314084848a4f28a042612f394 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23a1b73314084848a4f28a042612f394 already in progress: creating tablet
I20251024 19:48:11.232095 1774 raft_consensus.cc:2272] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:11.232260 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 28005301f9de48e5826f46c79aeb5fe5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28005301f9de48e5826f46c79aeb5fe5 already in progress: creating tablet
I20251024 19:48:11.232206 1186 catalog_manager.cc:5649] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } }
I20251024 19:48:11.232686 1186 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet a3259823b622461fa823841925b3d203 (table default.Table_ea0d3589c13a4ebeb950ddc737a9948b [id=350b37bf42194fcfbb660bffe83e47c4]). Expected version 4 got 0
I20251024 19:48:11.232975 1774 ts_tablet_manager.cc:1916] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:11.233518 1184 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_50e12cf5f39d4fa2983835ef0dbbc3d6 [id=201e5232763a4940a1288a56a695f71c] alter complete (version 1)
I20251024 19:48:11.233668 2640 raft_consensus.cc:493] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.233846 2640 raft_consensus.cc:515] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.234664 2640 leader_election.cc:290] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:11.235359 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "758df87acbdb4adaa83e1ecbc44e7cf1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:11.235533 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "758df87acbdb4adaa83e1ecbc44e7cf1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:11.235718 1510 raft_consensus.cc:2468] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:11.236136 1586 leader_election.cc:304] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:11.236451 1586 leader_election.cc:343] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.236951 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet ca0527af718f494a8bbcc66e7b325bed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.237030 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:11.237269 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet d76410afd7ba4c73836f4791ef739056 with delete_type TABLET_DATA_DELETED (replaced by tablet b153f75caebb4d869e77529c9c9bb50d) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.237390 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet 5fd7726c532145c6b44ea9a37544510d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.237663 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b96fc2309274048bed7a35e4bf4bae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.237794 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet b6661b9e9a174849b6093d09cd8c1251 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.238147 2640 raft_consensus.cc:2804] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:11.238230 2640 raft_consensus.cc:493] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:11.238286 2640 raft_consensus.cc:3060] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.238844 1766 tablet_bootstrap.cc:492] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:11.255828 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 46c1e2cef6cf4772b624060afb6fb117
I20251024 19:48:11.256484 1185 catalog_manager.cc:6719] ReplaceTablet: tablet 46c1e2cef6cf4772b624060afb6fb117 deleted and replaced by tablet da4315a1ac6948938e76da54924ce248
I20251024 19:48:11.257671 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_77837a284ba748f6852b42cc36890488"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:11.258194 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 46c1e2cef6cf4772b624060afb6fb117 with delete_type TABLET_DATA_DELETED (replaced by tablet da4315a1ac6948938e76da54924ce248) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:11.258268 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77837a284ba748f6852b42cc36890488 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.258714 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 46c1e2cef6cf4772b624060afb6fb117 with delete_type TABLET_DATA_DELETED (replaced by tablet da4315a1ac6948938e76da54924ce248) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:11.260440 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 25ced70ff40d4149bb92e8b3767ec38f consensus state has no leader
I20251024 19:48:11.264652 1767 tablet_bootstrap.cc:492] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:11.266027 2567 raft_consensus.cc:493] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af3952de050140d383f6b0de7666fa97)
I20251024 19:48:11.266192 2567 raft_consensus.cc:515] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.278689 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet 6f7b6d15f25d4c2a905da6597ba8d98a from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:11.279147 1447 leader_election.cc:343] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.280525 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 913534ab88dd459e9cac2faebb5492a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.281224 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 46c1e2cef6cf4772b624060afb6fb117 with delete_type TABLET_DATA_DELETED (replaced by tablet da4315a1ac6948938e76da54924ce248) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:11.283967 1447 consensus_peers.cc:597] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd52295b14b14e70bbdcb45f1bd849e4. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:11.284125 1447 consensus_peers.cc:597] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:11.284360 1767 tablet_bootstrap.cc:654] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.286324 2567 leader_election.cc:290] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.286501 1768 tablet_bootstrap.cc:492] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:11.286646 1768 ts_tablet_manager.cc:1403] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.094s user 0.003s sys 0.000s
I20251024 19:48:11.287336 1768 raft_consensus.cc:359] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.287519 1768 raft_consensus.cc:385] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.287567 1768 raft_consensus.cc:740] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.287771 1768 consensus_queue.cc:260] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.288213 1768 ts_tablet_manager.cc:1434] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251024 19:48:11.288681 1768 tablet_bootstrap.cc:492] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:11.289916 1444 leader_election.cc:343] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: b6661b9e9a174849b6093d09cd8c1251
I20251024 19:48:11.291186 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b6661b9e9a174849b6093d09cd8c1251" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:11.291378 1364 raft_consensus.cc:2468] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:11.291596 1768 tablet_bootstrap.cc:654] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.291925 1447 leader_election.cc:304] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:11.292352 2567 raft_consensus.cc:2804] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251024 19:48:11.292433 2567 raft_consensus.cc:493] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election (detected failure of leader af3952de050140d383f6b0de7666fa97)
I20251024 19:48:11.292503 2567 raft_consensus.cc:3060] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Advancing to term 2
I20251024 19:48:11.298067 1650 raft_consensus.cc:2468] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:11.298168 1776 ts_tablet_manager.cc:1929] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:11.298295 1776 log.cc:1199] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/2037fcca87b94b5fb31639b4d07307b0
I20251024 19:48:11.298655 1776 ts_tablet_manager.cc:1950] T 2037fcca87b94b5fb31639b4d07307b0 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:11.309284 1444 leader_election.cc:304] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:11.309509 1776 tablet_replica.cc:333] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:11.309743 1776 raft_consensus.cc:2243] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:11.309739 2561 raft_consensus.cc:2804] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:11.309854 2561 raft_consensus.cc:697] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:11.309943 1776 raft_consensus.cc:2272] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:11.310066 2561 consensus_queue.cc:237] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.310461 1776 ts_tablet_manager.cc:1916] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:11.312769 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 2037fcca87b94b5fb31639b4d07307b0 (table default.Table_a4b63032d2224a6d95765cb93137f4a2 [id=d28fd80220c64f8496a09f5d4e712874]) successfully deleted
W20251024 19:48:11.313061 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3da81c18d9264f38966944ad13f256d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3da81c18d9264f38966944ad13f256d8 already in progress: creating tablet
W20251024 19:48:11.313375 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 250f106664be4b5aaa330dbb4e43cb7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 250f106664be4b5aaa330dbb4e43cb7b already in progress: creating tablet
W20251024 19:48:11.313726 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 23a1b73314084848a4f28a042612f394 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23a1b73314084848a4f28a042612f394 already in progress: creating tablet
I20251024 19:48:11.314078 1774 ts_tablet_manager.cc:1929] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20251024 19:48:11.314106 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 28005301f9de48e5826f46c79aeb5fe5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28005301f9de48e5826f46c79aeb5fe5 already in progress: creating tablet
I20251024 19:48:11.314190 1774 log.cc:1199] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/913534ab88dd459e9cac2faebb5492a1
I20251024 19:48:11.314585 1774 ts_tablet_manager.cc:1950] T 913534ab88dd459e9cac2faebb5492a1 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:48:11.315904 1444 consensus_peers.cc:597] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd52295b14b14e70bbdcb45f1bd849e4. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:11.316334 2567 raft_consensus.cc:515] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.316897 2567 leader_election.cc:290] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:11.320924 1444 leader_election.cc:343] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: b6661b9e9a174849b6093d09cd8c1251
I20251024 19:48:11.321447 1767 tablet_bootstrap.cc:492] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:11.321630 1767 ts_tablet_manager.cc:1403] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.001s
I20251024 19:48:11.322005 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b6661b9e9a174849b6093d09cd8c1251" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:11.322260 1364 raft_consensus.cc:3060] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 2
I20251024 19:48:11.322415 1767 raft_consensus.cc:359] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.322576 1767 raft_consensus.cc:385] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.322624 1767 raft_consensus.cc:740] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.322865 1767 consensus_queue.cc:260] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.324043 1767 ts_tablet_manager.cc:1434] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:11.324327 1767 tablet_bootstrap.cc:492] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:11.327044 2656 raft_consensus.cc:493] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.327162 2656 raft_consensus.cc:515] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.327746 2656 leader_election.cc:290] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:11.329396 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3259823b622461fa823841925b3d203" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:11.331161 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3259823b622461fa823841925b3d203" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:11.331403 1510 raft_consensus.cc:2393] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:11.331987 1308 leader_election.cc:304] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:11.332480 2656 raft_consensus.cc:3060] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.340404 1183 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_b09229e0e1254adeacfed952befaa65a [id=67a0be12c85648bdb4515a096db21d3b] alter complete (version 1)
I20251024 19:48:11.343262 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f186552a977c4a1e8002ece18abd9d07" } new_table_name: "default.Table_082186ad39f14242b8dbcf092a97ae7d" modify_external_catalogs: true
I20251024 19:48:11.345232 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b5a5283ea815484a91f644c13a1826b7" } new_table_name: "default.Table_6b8a2c3a14404cab96880c1a8630f56c" modify_external_catalogs: true
I20251024 19:48:11.345340 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0f16fa751e6244afbe2f1683e608bc81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:11.346065 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f16fa751e6244afbe2f1683e608bc81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.346371 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 7352d84226db467dad31ae5941842b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.346669 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet b79801594e95434cbb8fe7b75b14602e with delete_type TABLET_DATA_DELETED (replaced by tablet 3e6f8764e9ae472b9ac35a032182aed1) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.346926 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet fdfa5eea3b674a3389a156d1c72204df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.347043 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 59c7dbe9ff2c4869994c4f43d5676c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.347105 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fd8aa1d9aca429dbfe46f472ebbf53b) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.347551 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet 2037fcca87b94b5fb31639b4d07307b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d129bdc876e425aabcf1c99997e319a) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.347834 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet dc14b0cdb7f342ac956a7857ab3f35d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.353530 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9
I20251024 19:48:11.354132 1184 catalog_manager.cc:6719] ReplaceTablet: tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 deleted and replaced by tablet a64e3591bc544283b409a0666fbcd865
I20251024 19:48:11.354393 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 with delete_type TABLET_DATA_DELETED (replaced by tablet a64e3591bc544283b409a0666fbcd865) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.355263 1767 tablet_bootstrap.cc:654] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.355809 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 with delete_type TABLET_DATA_DELETED (replaced by tablet a64e3591bc544283b409a0666fbcd865) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.355603 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 with delete_type TABLET_DATA_DELETED (replaced by tablet a64e3591bc544283b409a0666fbcd865) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.359030 1768 tablet_bootstrap.cc:492] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:11.359246 1768 ts_tablet_manager.cc:1403] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.000s
I20251024 19:48:11.360009 1768 raft_consensus.cc:359] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.360286 1768 raft_consensus.cc:385] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.360390 1768 raft_consensus.cc:740] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.360671 1768 consensus_queue.cc:260] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.366979 1768 ts_tablet_manager.cc:1434] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251024 19:48:11.370047 1766 tablet_bootstrap.cc:654] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.371148 1768 tablet_bootstrap.cc:492] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:11.375674 1192 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_f203b363de4c47fca0a0599140d18279 [id=ec57efb4d37e44f1b68045f58bc11c45] alter complete (version 1)
I20251024 19:48:11.376094 1774 tablet_replica.cc:333] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:11.376402 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]) successfully deleted
I20251024 19:48:11.376459 1774 raft_consensus.cc:2243] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:11.376609 1774 raft_consensus.cc:2272] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:11.376879 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 2037fcca87b94b5fb31639b4d07307b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d129bdc876e425aabcf1c99997e319a) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.377161 1774 ts_tablet_manager.cc:1916] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:11.377898 1768 tablet_bootstrap.cc:654] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.378661 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet b79801594e95434cbb8fe7b75b14602e with delete_type TABLET_DATA_DELETED (replaced by tablet 3e6f8764e9ae472b9ac35a032182aed1) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.379434 1767 tablet_bootstrap.cc:492] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:11.379593 1767 ts_tablet_manager.cc:1403] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.000s
I20251024 19:48:11.380357 1767 raft_consensus.cc:359] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.380579 1767 raft_consensus.cc:385] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.380662 1767 raft_consensus.cc:740] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.380903 1767 consensus_queue.cc:260] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.384567 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet dc14b0cdb7f342ac956a7857ab3f35d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.385558 1767 ts_tablet_manager.cc:1434] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251024 19:48:11.386670 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6811639fccc645fd9ccd5bfe038fbadd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:11.386935 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 73ffe342f541447d8519ee24bc98a0f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 505e4fb0755a4caaa462e5f5154bfee2) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.387130 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet ad81d80697984ff1834a7d795e0f4b9b with delete_type TABLET_DATA_DELETED (replaced by tablet 7c880a4af94d4041b8c7cf2066002ee3) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.387264 1767 tablet_bootstrap.cc:492] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:11.387305 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 7352d84226db467dad31ae5941842b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:11.387331 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6811639fccc645fd9ccd5bfe038fbadd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.389259 1372 raft_consensus.cc:3060] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.386806 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7ab38232d57d40fea8917ef3a67df1ce" } modify_external_catalogs: true
I20251024 19:48:11.389959 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7ab38232d57d40fea8917ef3a67df1ce" } modify_external_catalogs: true
I20251024 19:48:11.393692 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b89a90bb4ca447fbaf34fe39001c261 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.393977 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fd8aa1d9aca429dbfe46f472ebbf53b) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.394658 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_81ca5ba9d12c41cd9067ca6fef94e0b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:11.394881 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet fdfa5eea3b674a3389a156d1c72204df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.394826 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e87be853efba49aeb50e71270beac600" } new_table_name: "default.Table_c737c6e46264439f8401a4a81d1fa4ae" modify_external_catalogs: true
W20251024 19:48:11.395634 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81ca5ba9d12c41cd9067ca6fef94e0b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.397119 1768 tablet_bootstrap.cc:492] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:11.397262 1768 ts_tablet_manager.cc:1403] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20251024 19:48:11.398430 2638 raft_consensus.cc:515] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.397543 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c3cc8727161245ab88c9c67c4b45546f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:11.399053 2638 leader_election.cc:290] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:11.400820 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3cc8727161245ab88c9c67c4b45546f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.398000 1768 raft_consensus.cc:359] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.406183 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ac4ce453efea4a268f70380f9ed99812" } modify_external_catalogs: true
I20251024 19:48:11.406235 1768 raft_consensus.cc:385] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.406415 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ac4ce453efea4a268f70380f9ed99812" } modify_external_catalogs: true
I20251024 19:48:11.407510 1767 tablet_bootstrap.cc:654] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.407931 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bf08ccf7054fedb9efcbb00354c71a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:11.408118 1510 raft_consensus.cc:3060] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.409761 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bf08ccf7054fedb9efcbb00354c71a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:11.411787 2615 raft_consensus.cc:493] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.411931 2615 raft_consensus.cc:515] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.412575 2615 leader_election.cc:290] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:11.415179 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9377725db312439e8a42dbd46da84a9d consensus state has no leader
W20251024 19:48:11.415700 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3e9960b7821421aa23e7ef5389f894a consensus state has no leader
W20251024 19:48:11.420861 1586 leader_election.cc:343] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.421149 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b89a90bb4ca447fbaf34fe39001c261" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:11.421362 1650 raft_consensus.cc:2468] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:11.422011 1444 leader_election.cc:304] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:11.422364 2561 raft_consensus.cc:2804] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:11.422434 2561 raft_consensus.cc:493] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:11.422499 2561 raft_consensus.cc:3060] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.422920 1774 ts_tablet_manager.cc:1929] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:11.423034 1774 log.cc:1199] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/ca0527af718f494a8bbcc66e7b325bed
I20251024 19:48:11.423310 1774 ts_tablet_manager.cc:1950] T ca0527af718f494a8bbcc66e7b325bed P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:11.423238 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b89a90bb4ca447fbaf34fe39001c261" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:11.423642 1776 ts_tablet_manager.cc:1929] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:11.423744 1776 log.cc:1199] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/913534ab88dd459e9cac2faebb5492a1
W20251024 19:48:11.423933 1447 leader_election.cc:343] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.424247 1364 raft_consensus.cc:2468] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 2.
I20251024 19:48:11.424537 1776 ts_tablet_manager.cc:1950] T 913534ab88dd459e9cac2faebb5492a1 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:11.406523 1768 raft_consensus.cc:740] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.427446 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3f7f7b5dfa674b19ad19c339c619326c" } modify_external_catalogs: true
I20251024 19:48:11.427675 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3f7f7b5dfa674b19ad19c339c619326c" } modify_external_catalogs: true
I20251024 19:48:11.428253 1447 leader_election.cc:304] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:11.428323 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_405cbdb6a024494a81e147548e24d436" } new_table_name: "default.Table_a8e5ad4c50ba487c90b7e49a84c41357" modify_external_catalogs: true
I20251024 19:48:11.428750 2567 raft_consensus.cc:2804] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Leader election won for term 2
I20251024 19:48:11.428864 2567 raft_consensus.cc:697] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 2 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:11.429081 2567 consensus_queue.cc:237] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.429625 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_345aac1447b347f99accafba7777859a" } modify_external_catalogs: true
I20251024 19:48:11.429772 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_345aac1447b347f99accafba7777859a" } modify_external_catalogs: true
W20251024 19:48:11.434185 1177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.252.254:36505: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20251024 19:48:11.434971 1766 tablet_bootstrap.cc:492] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:11.435110 1766 ts_tablet_manager.cc:1403] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.196s user 0.005s sys 0.000s
W20251024 19:48:11.435617 1177 consensus_peers.cc:597] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c -> Peer ef8906492b1a4368936fac95c11e1f73 (127.0.252.254:36505): Couldn't send request to peer ef8906492b1a4368936fac95c11e1f73. Status: Network error: Client connection negotiation failed: client connection to 127.0.252.254:36505: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251024 19:48:11.435887 1766 raft_consensus.cc:359] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.425724 1768 consensus_queue.cc:260] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.436050 1766 raft_consensus.cc:385] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.436126 1766 raft_consensus.cc:740] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.436328 1766 consensus_queue.cc:260] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.436590 1768 ts_tablet_manager.cc:1434] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.039s user 0.003s sys 0.000s
I20251024 19:48:11.436832 1766 ts_tablet_manager.cc:1434] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:11.437110 1766 tablet_bootstrap.cc:492] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:11.440148 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6471c185917e4751962898ec47aa725b" } modify_external_catalogs: true
I20251024 19:48:11.440326 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6471c185917e4751962898ec47aa725b" } modify_external_catalogs: true
I20251024 19:48:11.443923 1766 tablet_bootstrap.cc:654] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.446496 2663 tablet.cc:1722] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:11.449978 1364 raft_consensus.cc:3060] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.452821 2610 tablet.cc:1722] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:11.452852 2675 tablet.cc:1722] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:11.457795 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 7352d84226db467dad31ae5941842b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.458667 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1ea2bdfecf104bf18dacacebb8b5dcc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:11.459416 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ea2bdfecf104bf18dacacebb8b5dcc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.466122 1768 tablet_bootstrap.cc:492] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:11.466569 2638 consensus_queue.cc:1048] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:11.471228 1768 tablet_bootstrap.cc:654] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:11.474670 1586 consensus_peers.cc:597] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:11.478039 1767 tablet_bootstrap.cc:492] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:11.478178 1767 ts_tablet_manager.cc:1403] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.091s user 0.000s sys 0.004s
I20251024 19:48:11.479001 1767 raft_consensus.cc:359] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.479158 1767 raft_consensus.cc:385] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.479205 1767 raft_consensus.cc:740] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.479418 1767 consensus_queue.cc:260] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.480868 1767 ts_tablet_manager.cc:1434] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:11.495224 2640 raft_consensus.cc:515] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.495318 1775 ts_tablet_manager.cc:1929] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:11.495446 1775 log.cc:1199] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/8a1f1892bd3b4776adc7c2519fcdf411
I20251024 19:48:11.495790 1775 ts_tablet_manager.cc:1950] T 8a1f1892bd3b4776adc7c2519fcdf411 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:11.495967 2640 leader_election.cc:290] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:11.496003 1510 raft_consensus.cc:2468] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:11.498611 1776 tablet_replica.cc:333] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:11.498842 1776 raft_consensus.cc:2243] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:11.499004 1776 raft_consensus.cc:2272] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:11.499308 2656 raft_consensus.cc:2749] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:11.499513 1776 ts_tablet_manager.cc:1916] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:11.499825 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "758df87acbdb4adaa83e1ecbc44e7cf1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:11.501355 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "758df87acbdb4adaa83e1ecbc44e7cf1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:11.501552 1510 raft_consensus.cc:3060] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.502521 1586 leader_election.cc:304] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:11.502630 1767 tablet_bootstrap.cc:492] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:11.503796 1774 tablet_replica.cc:333] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:11.503968 1774 raft_consensus.cc:2243] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:11.504101 1774 raft_consensus.cc:2272] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:11.504606 1774 ts_tablet_manager.cc:1916] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:11.506268 1586 leader_election.cc:343] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.508534 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet ca0527af718f494a8bbcc66e7b325bed (table default.Table_51982d999b0847ec87fc62a473782a5c [id=9a05bcf7ad5d430c9ef6d6e18a27808e]) successfully deleted
W20251024 19:48:11.509294 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b2992a738b440d798f5e61cc45ec088 consensus state has no leader
I20251024 19:48:11.510975 2569 raft_consensus.cc:2804] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:11.511340 2638 raft_consensus.cc:493] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.511461 2638 raft_consensus.cc:515] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.511427 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_43dbb020b49b44cfa627ef85edba938c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:11.512049 2638 leader_election.cc:290] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.512420 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 701c93a600964c20a275f5942a2ff863 with delete_type TABLET_DATA_DELETED (replaced by tablet 7da1601a87524ef1a8b5069ba6823843) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.513335 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc14b0cdb7f342ac956a7857ab3f35d7" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:11.514000 1586 leader_election.cc:343] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.518232 2640 raft_consensus.cc:493] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.518395 2640 raft_consensus.cc:515] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.519045 2640 leader_election.cc:290] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.520314 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03cdbd1b62b741249ad78562a7edfd22" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:11.520867 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 635a132735704cfda69bf7c8deaea81d (table default.Table_933a6697624149369ab3ec28462b466a [id=93a61862707b4a7ba8a0ff349704dc2b]). Expected version 1 got 0
W20251024 19:48:11.520937 1586 leader_election.cc:343] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.524133 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc14b0cdb7f342ac956a7857ab3f35d7" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:11.524336 1508 raft_consensus.cc:2468] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:11.524582 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03cdbd1b62b741249ad78562a7edfd22" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:11.525126 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_645f2377cbe34c8dbacd9c416c00cfa1" } new_table_name: "default.Table_8839c58c3b974b41938ec5bba438af46" modify_external_catalogs: true
I20251024 19:48:11.525560 1586 leader_election.cc:304] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:11.525836 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]) successfully deleted
I20251024 19:48:11.525941 1768 tablet_bootstrap.cc:492] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
W20251024 19:48:11.526019 1586 leader_election.cc:343] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.526075 1768 ts_tablet_manager.cc:1403] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.000s
W20251024 19:48:11.526100 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 46c1e2cef6cf4772b624060afb6fb117 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46c1e2cef6cf4772b624060afb6fb117 already in progress: creating tablet
I20251024 19:48:11.526122 1586 leader_election.cc:304] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:11.526652 2638 raft_consensus.cc:2804] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:11.526736 2638 raft_consensus.cc:493] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:11.526799 2638 raft_consensus.cc:3060] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.527208 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 83d34ec47c2845058e870c72cc832e9e (DEFAULT_TABLE table=default.Table_ec9fec2116ab4e40bff85f19b11d6f73 [id=64dceb7b69034613bfcf74959411870f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.527273 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 08b1b461a5de434d83ba5b6c4587a66f (DEFAULT_TABLE table=default.Table_075cb16ace064e7a8840d0d6d2ede7ec [id=99b3e987fe814cc9ac7b2cd127a61f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.527532 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 82cf75ee148e45b29da3343b3b068781 (DEFAULT_TABLE table=default.Table_6471c185917e4751962898ec47aa725b [id=13c6b2364fc44ee4ba524e15dd76c4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.527642 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08b1b461a5de434d83ba5b6c4587a66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.527841 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82cf75ee148e45b29da3343b3b068781. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.527860 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 8a98058c346f4bb89d07f7409052aade (DEFAULT_TABLE table=default.Table_34c8fbbe53954b97b7ead671ade3783b [id=4be1da24847842b596cd84b5c1ca1162]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.528167 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a98058c346f4bb89d07f7409052aade. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.531965 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d34ec47c2845058e870c72cc832e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.533701 2640 raft_consensus.cc:2749] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:11.533996 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43dbb020b49b44cfa627ef85edba938c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.534269 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 08b1b461a5de434d83ba5b6c4587a66f (DEFAULT_TABLE table=default.Table_075cb16ace064e7a8840d0d6d2ede7ec [id=99b3e987fe814cc9ac7b2cd127a61f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.534682 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08b1b461a5de434d83ba5b6c4587a66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.535187 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 187925082cdd4923a6e9148bf9f270fd (table default.Table_55f8c66bd97343438051a98af12d8c2c [id=33f5ac73298543d0b5af77a27e897b79]). Expected version 1 got 0
I20251024 19:48:11.535880 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 83d34ec47c2845058e870c72cc832e9e (DEFAULT_TABLE table=default.Table_ec9fec2116ab4e40bff85f19b11d6f73 [id=64dceb7b69034613bfcf74959411870f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.536211 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d34ec47c2845058e870c72cc832e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.536616 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 82cf75ee148e45b29da3343b3b068781 (DEFAULT_TABLE table=default.Table_6471c185917e4751962898ec47aa725b [id=13c6b2364fc44ee4ba524e15dd76c4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.536927 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82cf75ee148e45b29da3343b3b068781. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.537833 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 8a98058c346f4bb89d07f7409052aade (DEFAULT_TABLE table=default.Table_34c8fbbe53954b97b7ead671ade3783b [id=4be1da24847842b596cd84b5c1ca1162]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.538933 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 08b1b461a5de434d83ba5b6c4587a66f (DEFAULT_TABLE table=default.Table_075cb16ace064e7a8840d0d6d2ede7ec [id=99b3e987fe814cc9ac7b2cd127a61f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.539347 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08b1b461a5de434d83ba5b6c4587a66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.539702 1349 tablet_service.cc:1505] Processing CreateTablet for tablet e7d6472ac7b046d89f56618fa4ed1811 (DEFAULT_TABLE table=default.Table_c55267a23ded43cab7442fa75b611e12 [id=99470fb60280469b88018a292ea80c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.540309 1475 tablet_service.cc:1505] Processing CreateTablet for tablet e7d6472ac7b046d89f56618fa4ed1811 (DEFAULT_TABLE table=default.Table_c55267a23ded43cab7442fa75b611e12 [id=99470fb60280469b88018a292ea80c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.540628 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7d6472ac7b046d89f56618fa4ed1811. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.540980 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bf08ccf7054fedb9efcbb00354c71a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.541177 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b89a90bb4ca447fbaf34fe39001c261 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.541642 1627 tablet_service.cc:1505] Processing CreateTablet for tablet e7d6472ac7b046d89f56618fa4ed1811 (DEFAULT_TABLE table=default.Table_c55267a23ded43cab7442fa75b611e12 [id=99470fb60280469b88018a292ea80c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.541841 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 82cf75ee148e45b29da3343b3b068781 (DEFAULT_TABLE table=default.Table_6471c185917e4751962898ec47aa725b [id=13c6b2364fc44ee4ba524e15dd76c4d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.542187 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82cf75ee148e45b29da3343b3b068781. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.545552 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 8a98058c346f4bb89d07f7409052aade (DEFAULT_TABLE table=default.Table_34c8fbbe53954b97b7ead671ade3783b [id=4be1da24847842b596cd84b5c1ca1162]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.545935 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a98058c346f4bb89d07f7409052aade. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.547050 1617 tablet_service.cc:1505] Processing CreateTablet for tablet e0b57403cdf947b390a44a9889f2a2c3 (DEFAULT_TABLE table=default.Table_dcb0f784408346bda089b634ed26412e [id=28e8b430b0b64b188d5607934980af26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.547430 1487 tablet_service.cc:1505] Processing CreateTablet for tablet e0b57403cdf947b390a44a9889f2a2c3 (DEFAULT_TABLE table=default.Table_dcb0f784408346bda089b634ed26412e [id=28e8b430b0b64b188d5607934980af26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.547725 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0b57403cdf947b390a44a9889f2a2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.548378 1767 tablet_bootstrap.cc:654] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.550230 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 83d34ec47c2845058e870c72cc832e9e (DEFAULT_TABLE table=default.Table_ec9fec2116ab4e40bff85f19b11d6f73 [id=64dceb7b69034613bfcf74959411870f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.550552 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d34ec47c2845058e870c72cc832e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.552404 1616 tablet_service.cc:1505] Processing CreateTablet for tablet c12881bc415941e4a236256fd0b74a90 (DEFAULT_TABLE table=default.Table_49ca74f7a5d24c139a894b1bb06f7f5d [id=8a6d9c617c3649c58791939fb879167a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.553493 1486 tablet_service.cc:1505] Processing CreateTablet for tablet c12881bc415941e4a236256fd0b74a90 (DEFAULT_TABLE table=default.Table_49ca74f7a5d24c139a894b1bb06f7f5d [id=8a6d9c617c3649c58791939fb879167a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.553889 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12881bc415941e4a236256fd0b74a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.554328 1620 tablet_service.cc:1505] Processing CreateTablet for tablet e3405f63ee04404298b1f0e0527522e3 (DEFAULT_TABLE table=default.Table_645f2377cbe34c8dbacd9c416c00cfa1 [id=9d2ab76033e044f9999c671ec0be3de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.554445 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 82cf75ee148e45b29da3343b3b068781
I20251024 19:48:11.555261 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 82cf75ee148e45b29da3343b3b068781 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.556384 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4dca2888019448ca97c68a8c6dd8396a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:11.556999 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dca2888019448ca97c68a8c6dd8396a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.557224 1483 tablet_service.cc:1505] Processing CreateTablet for tablet e3405f63ee04404298b1f0e0527522e3 (DEFAULT_TABLE table=default.Table_645f2377cbe34c8dbacd9c416c00cfa1 [id=9d2ab76033e044f9999c671ec0be3de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.557403 1768 raft_consensus.cc:359] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.557543 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3405f63ee04404298b1f0e0527522e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.557626 1768 raft_consensus.cc:385] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.557694 1768 raft_consensus.cc:740] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.557895 1768 consensus_queue.cc:260] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.558385 1768 ts_tablet_manager.cc:1434] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20251024 19:48:11.560559 1619 tablet_service.cc:1552] Processing DeleteTablet for tablet 82cf75ee148e45b29da3343b3b068781 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.562077 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet 82cf75ee148e45b29da3343b3b068781 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.562228 1343 tablet_service.cc:1505] Processing CreateTablet for tablet e0b57403cdf947b390a44a9889f2a2c3 (DEFAULT_TABLE table=default.Table_dcb0f784408346bda089b634ed26412e [id=28e8b430b0b64b188d5607934980af26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.562749 1344 tablet_service.cc:1505] Processing CreateTablet for tablet c12881bc415941e4a236256fd0b74a90 (DEFAULT_TABLE table=default.Table_49ca74f7a5d24c139a894b1bb06f7f5d [id=8a6d9c617c3649c58791939fb879167a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.563231 1345 tablet_service.cc:1505] Processing CreateTablet for tablet e3405f63ee04404298b1f0e0527522e3 (DEFAULT_TABLE table=default.Table_645f2377cbe34c8dbacd9c416c00cfa1 [id=9d2ab76033e044f9999c671ec0be3de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.563565 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 8876b7b38e744212a3722dbe5bcc5596
I20251024 19:48:11.563680 1341 tablet_service.cc:1505] Processing CreateTablet for tablet a0061fee0c324725a01cedb727636831 (DEFAULT_TABLE table=default.Table_077b6c98744f4a5baa42d5c4bd0da645 [id=5e58bddeb36c4ab3a24f3eaccf1114ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.562794 1336 tablet_service.cc:1505] Processing CreateTablet for tablet b724eadbb86e4eeab4ba8dfd565290ff (DEFAULT_TABLE table=default.Table_81a931b6624742858d05cfc15a78beed [id=32262872cb9442a6a69bf4aed3c43b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.564229 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet 452f77091adc4cf09de2231f2f6d845b from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:11.564469 1619 tablet_service.cc:1505] Processing CreateTablet for tablet a0061fee0c324725a01cedb727636831 (DEFAULT_TABLE table=default.Table_077b6c98744f4a5baa42d5c4bd0da645 [id=5e58bddeb36c4ab3a24f3eaccf1114ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.564551 2569 raft_consensus.cc:697] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:11.564851 2569 consensus_queue.cc:237] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.565207 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 8876b7b38e744212a3722dbe5bcc5596 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.565995 1346 tablet_service.cc:1505] Processing CreateTablet for tablet f0b1e072db9742c38ce3b09bd195dcc0 (DEFAULT_TABLE table=default.Table_9ee642f4c0344db6b73e09e30ab6d351 [id=30ded4999d5642e6ab14def999ad4f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.566507 1479 tablet_service.cc:1505] Processing CreateTablet for tablet a0061fee0c324725a01cedb727636831 (DEFAULT_TABLE table=default.Table_077b6c98744f4a5baa42d5c4bd0da645 [id=5e58bddeb36c4ab3a24f3eaccf1114ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.566794 1630 tablet_service.cc:1505] Processing CreateTablet for tablet b724eadbb86e4eeab4ba8dfd565290ff (DEFAULT_TABLE table=default.Table_81a931b6624742858d05cfc15a78beed [id=32262872cb9442a6a69bf4aed3c43b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.566905 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0061fee0c324725a01cedb727636831. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.567435 1490 tablet_service.cc:1505] Processing CreateTablet for tablet f0b1e072db9742c38ce3b09bd195dcc0 (DEFAULT_TABLE table=default.Table_9ee642f4c0344db6b73e09e30ab6d351 [id=30ded4999d5642e6ab14def999ad4f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.567744 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0b1e072db9742c38ce3b09bd195dcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.568156 1767 tablet_bootstrap.cc:492] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:11.568274 1767 ts_tablet_manager.cc:1403] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.066s user 0.001s sys 0.002s
I20251024 19:48:11.569010 1767 raft_consensus.cc:359] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.569159 1767 raft_consensus.cc:385] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.569200 1767 raft_consensus.cc:740] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.569382 1767 consensus_queue.cc:260] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.569847 1767 ts_tablet_manager.cc:1434] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:11.569908 1489 tablet_service.cc:1505] Processing CreateTablet for tablet b724eadbb86e4eeab4ba8dfd565290ff (DEFAULT_TABLE table=default.Table_81a931b6624742858d05cfc15a78beed [id=32262872cb9442a6a69bf4aed3c43b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.570226 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b724eadbb86e4eeab4ba8dfd565290ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.571071 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet 8876b7b38e744212a3722dbe5bcc5596 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.571560 1622 tablet_service.cc:1505] Processing CreateTablet for tablet f0b1e072db9742c38ce3b09bd195dcc0 (DEFAULT_TABLE table=default.Table_9ee642f4c0344db6b73e09e30ab6d351 [id=30ded4999d5642e6ab14def999ad4f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.571899 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 8876b7b38e744212a3722dbe5bcc5596 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.574478 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet cb0405765c6146cbbb187af96a60be2e
I20251024 19:48:11.575733 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_135cfe439b8e4e44a4317938ff165eaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:11.576068 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet cb0405765c6146cbbb187af96a60be2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.576258 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet cb0405765c6146cbbb187af96a60be2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:11.576401 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_135cfe439b8e4e44a4317938ff165eaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.577970 1766 tablet_bootstrap.cc:492] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:11.578089 1766 ts_tablet_manager.cc:1403] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.141s user 0.001s sys 0.009s
I20251024 19:48:11.578145 1340 tablet_service.cc:1505] Processing CreateTablet for tablet ec0a8fc1b67b4314ad8d305a9d13a1b6 (DEFAULT_TABLE table=default.Table_7fa0ffaa80294f7eb87d8b82770bb44c [id=a1a40743ffa04686abfd04929160e195]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.578794 1766 raft_consensus.cc:359] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.578958 1766 raft_consensus.cc:385] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.579013 1766 raft_consensus.cc:740] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.579371 1766 consensus_queue.cc:260] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.579602 1482 tablet_service.cc:1505] Processing CreateTablet for tablet ec0a8fc1b67b4314ad8d305a9d13a1b6 (DEFAULT_TABLE table=default.Table_7fa0ffaa80294f7eb87d8b82770bb44c [id=a1a40743ffa04686abfd04929160e195]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.579946 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0a8fc1b67b4314ad8d305a9d13a1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.582682 1766 ts_tablet_manager.cc:1434] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:48:11.582962 1766 tablet_bootstrap.cc:492] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:11.584029 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet cb0405765c6146cbbb187af96a60be2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.585002 1615 tablet_service.cc:1505] Processing CreateTablet for tablet ec0a8fc1b67b4314ad8d305a9d13a1b6 (DEFAULT_TABLE table=default.Table_7fa0ffaa80294f7eb87d8b82770bb44c [id=a1a40743ffa04686abfd04929160e195]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.587437 1768 tablet_bootstrap.cc:492] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:11.589237 2640 raft_consensus.cc:493] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.589419 2640 raft_consensus.cc:515] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.590241 2640 leader_election.cc:290] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.591112 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b88a6a5237d146f5b6f082ca3af25b2b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:11.591111 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b88a6a5237d146f5b6f082ca3af25b2b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:11.591782 1586 leader_election.cc:343] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:11.591969 1586 leader_election.cc:343] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.592060 1586 leader_election.cc:304] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:11.592468 2640 raft_consensus.cc:2749] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:11.593088 1766 tablet_bootstrap.cc:654] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.593755 1768 tablet_bootstrap.cc:654] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.597034 2561 raft_consensus.cc:515] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.597714 2561 leader_election.cc:290] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.598982 2638 raft_consensus.cc:515] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.599558 2638 leader_election.cc:290] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.600394 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7d6472ac7b046d89f56618fa4ed1811. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.600601 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0061fee0c324725a01cedb727636831. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.600795 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0a8fc1b67b4314ad8d305a9d13a1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.602651 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a98058c346f4bb89d07f7409052aade. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.603219 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0b57403cdf947b390a44a9889f2a2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.603298 1767 tablet_bootstrap.cc:492] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:11.603773 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12881bc415941e4a236256fd0b74a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.604115 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3405f63ee04404298b1f0e0527522e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.604568 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0061fee0c324725a01cedb727636831. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.605266 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7d6472ac7b046d89f56618fa4ed1811. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.605710 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b724eadbb86e4eeab4ba8dfd565290ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.606134 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0b1e072db9742c38ce3b09bd195dcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.606249 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 486) took 5612 ms. Trace:
I20251024 19:48:11.606534 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0a8fc1b67b4314ad8d305a9d13a1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.606364 1775 rpcz_store.cc:276] 1024 19:48:05.993349 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:05.994196 (+ 847us) service_pool.cc:225] Handling call
1024 19:48:10.896199 (+4902003us) ts_tablet_manager.cc:1151] Deleting tablet 8a1f1892bd3b4776adc7c2519fcdf411
1024 19:48:10.896217 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:11.303741 (+407524us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:11.495190 (+191449us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:11.606214 (+111024us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4901846}
I20251024 19:48:11.606877 1775 tablet_replica.cc:333] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:11.607120 1775 raft_consensus.cc:2243] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:11.607285 1775 raft_consensus.cc:2272] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:11.607931 1775 ts_tablet_manager.cc:1916] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:11.609730 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0b57403cdf947b390a44a9889f2a2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.610188 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12881bc415941e4a236256fd0b74a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.610567 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3405f63ee04404298b1f0e0527522e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.610637 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b89a90bb4ca447fbaf34fe39001c261" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:11.610870 1650 raft_consensus.cc:3060] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.610961 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b724eadbb86e4eeab4ba8dfd565290ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.611303 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0b1e072db9742c38ce3b09bd195dcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.611893 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b89a90bb4ca447fbaf34fe39001c261" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:11.618187 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 8a1f1892bd3b4776adc7c2519fcdf411 (table default.Table_46b7ffb5bf6f45f2be4d1b242f43bfc9 [id=0cd6320a10064e9a8556836dfdb2a1dc]) successfully deleted
W20251024 19:48:11.618767 1447 leader_election.cc:343] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:11.619444 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 9377725db312439e8a42dbd46da84a9d (table default.Table_082186ad39f14242b8dbcf092a97ae7d [id=2d70ca56de444b8bbc638451232f591b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9377725db312439e8a42dbd46da84a9d consensus state has no leader
I20251024 19:48:11.635686 2561 raft_consensus.cc:493] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.635878 2561 raft_consensus.cc:515] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.636600 2561 leader_election.cc:290] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.637385 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "452f77091adc4cf09de2231f2f6d845b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:11.637631 1647 raft_consensus.cc:2468] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:11.638316 1444 leader_election.cc:304] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:11.640478 1767 tablet_bootstrap.cc:654] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.643637 2567 raft_consensus.cc:2804] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251024 19:48:11.643743 2567 raft_consensus.cc:493] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:11.643801 2567 raft_consensus.cc:3060] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Advancing to term 3
I20251024 19:48:11.652257 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc14b0cdb7f342ac956a7857ab3f35d7" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:11.652460 1511 raft_consensus.cc:3060] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.653836 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc14b0cdb7f342ac956a7857ab3f35d7" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:11.654085 1369 raft_consensus.cc:3060] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.655465 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "452f77091adc4cf09de2231f2f6d845b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:11.655711 1366 raft_consensus.cc:2468] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 2.
W20251024 19:48:11.661015 1447 consensus_peers.cc:597] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fffc42ddda14bd28800917ff2f0ced5. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:11.667037 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet cbd321c059e34834a6257fab5e6d14bb
I20251024 19:48:11.668004 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 6f7b6d15f25d4c2a905da6597ba8d98a
I20251024 19:48:11.668453 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 50cea450eda9485daba40bf875a1b90c
I20251024 19:48:11.668524 1187 catalog_manager.cc:6719] ReplaceTablet: tablet 6f7b6d15f25d4c2a905da6597ba8d98a deleted and replaced by tablet 520060a9adca4bb0b8616bcc006dce52
I20251024 19:48:11.668666 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet cbd321c059e34834a6257fab5e6d14bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.669283 1472 tablet_service.cc:1552] Processing DeleteTablet for tablet cbd321c059e34834a6257fab5e6d14bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.670061 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet 6a7e76a3cac443ec8d2ab466075aae6b from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:11.670794 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3405f63ee04404298b1f0e0527522e3 consensus state has no leader
I20251024 19:48:11.670899 1333 tablet_service.cc:1552] Processing DeleteTablet for tablet 50cea450eda9485daba40bf875a1b90c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.671110 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f7b6d15f25d4c2a905da6597ba8d98a with delete_type TABLET_DATA_DELETED (replaced by tablet 520060a9adca4bb0b8616bcc006dce52) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.672273 1627 tablet_service.cc:1552] Processing DeleteTablet for tablet 50cea450eda9485daba40bf875a1b90c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.672580 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet cbd321c059e34834a6257fab5e6d14bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.672905 1191 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_c737c6e46264439f8401a4a81d1fa4ae [id=5bc052b3e0f04699b4b391bcec33617d] alter complete (version 2)
I20251024 19:48:11.673408 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f7b6d15f25d4c2a905da6597ba8d98a with delete_type TABLET_DATA_DELETED (replaced by tablet 520060a9adca4bb0b8616bcc006dce52) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.677026 1480 tablet_service.cc:1552] Processing DeleteTablet for tablet 50cea450eda9485daba40bf875a1b90c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.677302 1472 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f7b6d15f25d4c2a905da6597ba8d98a with delete_type TABLET_DATA_DELETED (replaced by tablet 520060a9adca4bb0b8616bcc006dce52) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.681476 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ecaba24cb11e46e980659c47d6cd0c49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:11.682301 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecaba24cb11e46e980659c47d6cd0c49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.683027 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_df275b460e8245c8bc231bab1fb44630"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:11.683662 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df275b460e8245c8bc231bab1fb44630 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.683816 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_281f4fc881c345e7b86cdc489422126b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:11.684152 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_60d5ed8e9b3846a8bc33ffae6069dda2" } new_table_name: "default.Table_48f96278488c4706a702542059d1c575" modify_external_catalogs: true
W20251024 19:48:11.684327 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_281f4fc881c345e7b86cdc489422126b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.685473 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_092242c51bbb4695947456a8ea5513dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:11.686084 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_092242c51bbb4695947456a8ea5513dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.694151 1767 tablet_bootstrap.cc:492] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:11.694304 1767 ts_tablet_manager.cc:1403] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.092s user 0.004s sys 0.000s
I20251024 19:48:11.695264 1767 raft_consensus.cc:359] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.695436 1767 raft_consensus.cc:385] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.695492 1767 raft_consensus.cc:740] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.698873 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2608adc41cfa4b96bc13dee99a9fca7b" } new_table_name: "default.Table_f2cdf925c04543daa2bdb70bd9f67dbe" modify_external_catalogs: true
I20251024 19:48:11.699369 1510 raft_consensus.cc:2468] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:11.702744 1767 consensus_queue.cc:260] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.705808 1650 raft_consensus.cc:2468] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:11.707381 1767 ts_tablet_manager.cc:1434] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20251024 19:48:11.707831 1586 leader_election.cc:304] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:11.708621 1776 ts_tablet_manager.cc:1929] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:11.708746 1776 log.cc:1199] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/7352d84226db467dad31ae5941842b80
I20251024 19:48:11.709028 1776 ts_tablet_manager.cc:1950] T 7352d84226db467dad31ae5941842b80 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:11.706408 1444 leader_election.cc:304] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:11.713553 2615 raft_consensus.cc:2804] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:11.713722 2615 raft_consensus.cc:697] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:11.717911 1767 tablet_bootstrap.cc:492] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:11.723100 1767 tablet_bootstrap.cc:654] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.721750 2613 raft_consensus.cc:493] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.727067 2613 raft_consensus.cc:515] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.727921 2613 leader_election.cc:290] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.729135 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "701c93a600964c20a275f5942a2ff863" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:11.729385 1650 raft_consensus.cc:2468] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:11.729990 1774 ts_tablet_manager.cc:1929] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:11.736974 2709 raft_consensus.cc:2804] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:11.737241 2709 raft_consensus.cc:697] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:11.737499 2709 consensus_queue.cc:237] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.742416 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 452f77091adc4cf09de2231f2f6d845b
I20251024 19:48:11.743022 1184 catalog_manager.cc:6719] ReplaceTablet: tablet 452f77091adc4cf09de2231f2f6d845b deleted and replaced by tablet f709db684b6c49cbaeef3bf8d76f5828
I20251024 19:48:11.743399 1444 leader_election.cc:304] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:11.743474 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5c78cfb1ef614578accc5ec60079087b" } new_table_name: "default.Table_761f1d6d97dc46c28951c41e9c5082b3" modify_external_catalogs: true
I20251024 19:48:11.744866 2656 raft_consensus.cc:493] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.745015 2656 raft_consensus.cc:515] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.745980 1767 tablet_bootstrap.cc:492] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:11.746059 2613 raft_consensus.cc:493] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.746119 1767 ts_tablet_manager.cc:1403] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20251024 19:48:11.746194 2613 raft_consensus.cc:515] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.747213 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet 452f77091adc4cf09de2231f2f6d845b with delete_type TABLET_DATA_DELETED (replaced by tablet f709db684b6c49cbaeef3bf8d76f5828) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.751686 1480 tablet_service.cc:1552] Processing DeleteTablet for tablet 452f77091adc4cf09de2231f2f6d845b with delete_type TABLET_DATA_DELETED (replaced by tablet f709db684b6c49cbaeef3bf8d76f5828) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:11.751758 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 373aa543e1284fbc945ca802d9eddf59 consensus state has no leader
I20251024 19:48:11.752053 2561 raft_consensus.cc:2804] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:11.752133 2561 raft_consensus.cc:493] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:11.752198 2561 raft_consensus.cc:3060] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.752779 2613 leader_election.cc:290] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.754055 1472 tablet_service.cc:1552] Processing DeleteTablet for tablet 27b32126d04e480f8db7b2b20aba5873 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.751080 2615 consensus_queue.cc:237] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.757045 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7503b22d9e12481b9af96385b9d76e50" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:11.757309 1510 raft_consensus.cc:2393] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:11.757941 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdfa5eea3b674a3389a156d1c72204df" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:11.758183 1650 raft_consensus.cc:2468] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:11.758828 1444 leader_election.cc:304] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:11.759162 2613 raft_consensus.cc:2804] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:11.759253 2613 raft_consensus.cc:493] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:11.759317 2613 raft_consensus.cc:3060] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.761691 2610 tablet.cc:1722] T d692e1988d13460fb1d54e930e17fc9d P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:11.762527 1766 tablet_bootstrap.cc:492] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:11.762758 1766 ts_tablet_manager.cc:1403] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.180s user 0.060s sys 0.058s
I20251024 19:48:11.766063 1766 raft_consensus.cc:359] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.766321 1766 raft_consensus.cc:385] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.766402 1766 raft_consensus.cc:740] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.766641 1766 consensus_queue.cc:260] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.767354 1766 ts_tablet_manager.cc:1434] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251024 19:48:11.767743 1766 tablet_bootstrap.cc:492] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:11.769173 1766 tablet_bootstrap.cc:654] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.772529 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8488a13f677347b591b4efde0761451b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:11.773125 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8488a13f677347b591b4efde0761451b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.776005 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0fa2426ca885458797e98bedc44dec04" } new_table_name: "default.Table_5bfbf5d7f5764b96a95bf3ac0e0aba69" modify_external_catalogs: true
I20251024 19:48:11.777062 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_922f48f987904120a61a2eba3d270993"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:11.777601 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 6a7e76a3cac443ec8d2ab466075aae6b
W20251024 19:48:11.777915 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_922f48f987904120a61a2eba3d270993 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.778687 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_379c43ee27bd4c74ae00adf4df216fa5" } modify_external_catalogs: true
I20251024 19:48:11.778877 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_379c43ee27bd4c74ae00adf4df216fa5" } modify_external_catalogs: true
I20251024 19:48:11.779848 1767 raft_consensus.cc:359] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.780035 1767 raft_consensus.cc:385] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.780086 1767 raft_consensus.cc:740] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.780331 1186 catalog_manager.cc:6719] ReplaceTablet: tablet 6a7e76a3cac443ec8d2ab466075aae6b deleted and replaced by tablet 2764689179504e079924fde7e425dbcd
I20251024 19:48:11.780654 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_48f96278488c4706a702542059d1c575" } modify_external_catalogs: true
I20251024 19:48:11.780808 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_48f96278488c4706a702542059d1c575" } modify_external_catalogs: true
I20251024 19:48:11.782371 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_475f0aa37b7b4cff876223e3d8a0514c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:11.783425 1472 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a7e76a3cac443ec8d2ab466075aae6b with delete_type TABLET_DATA_DELETED (replaced by tablet 2764689179504e079924fde7e425dbcd) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.784454 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a7e76a3cac443ec8d2ab466075aae6b with delete_type TABLET_DATA_DELETED (replaced by tablet 2764689179504e079924fde7e425dbcd) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.786020 2567 raft_consensus.cc:515] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.786706 2567 leader_election.cc:290] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 3 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.787113 1775 ts_tablet_manager.cc:1929] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:11.787216 1775 log.cc:1199] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/a76d61c3464b41519d7657796bd853ff
I20251024 19:48:11.787514 1775 ts_tablet_manager.cc:1950] T a76d61c3464b41519d7657796bd853ff P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:11.789039 1511 raft_consensus.cc:2468] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:11.791152 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 7352d84226db467dad31ae5941842b80 (table default.Table_71bcb15d38b842efbdfc0fb90171e71e [id=d115a3523a5746918a343221a41c9589]) successfully deleted
I20251024 19:48:11.791663 1776 tablet_replica.cc:333] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:11.791908 1776 raft_consensus.cc:2243] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:11.792346 1586 leader_election.cc:304] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:11.792562 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "452f77091adc4cf09de2231f2f6d845b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:11.792717 2709 raft_consensus.cc:2804] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:11.792718 1650 raft_consensus.cc:3060] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Advancing to term 3
I20251024 19:48:11.792950 2709 raft_consensus.cc:697] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:11.793181 2709 consensus_queue.cc:237] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:11.793889 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 672b571081ea4cd78028b6a25625103c consensus state has no leader
I20251024 19:48:11.794255 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 481) took 5801 ms. Trace:
I20251024 19:48:11.794335 1775 rpcz_store.cc:276] 1024 19:48:05.992859 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.001554 (+ 8695us) service_pool.cc:225] Handling call
1024 19:48:11.606811 (+5605257us) ts_tablet_manager.cc:1151] Deleting tablet a76d61c3464b41519d7657796bd853ff
1024 19:48:11.606824 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:11.708780 (+101956us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:11.786995 (+ 78215us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:11.794233 (+ 7238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5594666}
I20251024 19:48:11.794670 1775 tablet_replica.cc:333] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:11.794868 1775 raft_consensus.cc:2243] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:11.795238 1775 raft_consensus.cc:2272] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:11.791841 1767 consensus_queue.cc:260] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.795697 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_aec0055f05e645e5a429d88125c4fb6f" } modify_external_catalogs: true
I20251024 19:48:11.795887 1775 ts_tablet_manager.cc:1916] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:11.795914 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_aec0055f05e645e5a429d88125c4fb6f" } modify_external_catalogs: true
I20251024 19:48:11.798149 1767 ts_tablet_manager.cc:1434] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.052s user 0.003s sys 0.000s
I20251024 19:48:11.800647 1776 raft_consensus.cc:2272] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:11.801688 1776 ts_tablet_manager.cc:1916] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:11.802881 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_475f0aa37b7b4cff876223e3d8a0514c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:11.808360 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca6b8783039a4f5fbbda65c129805642 consensus state has no leader
I20251024 19:48:11.808784 1767 tablet_bootstrap.cc:492] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:11.810384 1767 tablet_bootstrap.cc:654] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.811972 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a7e76a3cac443ec8d2ab466075aae6b with delete_type TABLET_DATA_DELETED (replaced by tablet 2764689179504e079924fde7e425dbcd) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.812300 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 452f77091adc4cf09de2231f2f6d845b with delete_type TABLET_DATA_DELETED (replaced by tablet f709db684b6c49cbaeef3bf8d76f5828) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:11.812851 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7503b22d9e12481b9af96385b9d76e50" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:11.745736 2656 leader_election.cc:290] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:11.816466 1774 log.cc:1199] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/d76410afd7ba4c73836f4791ef739056
I20251024 19:48:11.818727 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "701c93a600964c20a275f5942a2ff863" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:11.819422 1372 raft_consensus.cc:3060] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.823308 2567 raft_consensus.cc:493] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.823504 2567 raft_consensus.cc:515] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.816810 1774 ts_tablet_manager.cc:1950] T d76410afd7ba4c73836f4791ef739056 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:11.824122 2567 leader_election.cc:290] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:11.825237 1447 leader_election.cc:343] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.827204 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet b88a6a5237d146f5b6f082ca3af25b2b with delete_type TABLET_DATA_DELETED (replaced by tablet 639836c656f44a819036f4d00e055bd9) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:11.827637 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b88a6a5237d146f5b6f082ca3af25b2b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:11.827813 1647 raft_consensus.cc:2468] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:11.829061 1444 leader_election.cc:304] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:11.829313 2615 raft_consensus.cc:2804] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:11.829380 2615 raft_consensus.cc:493] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:11.829448 2615 raft_consensus.cc:3060] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.834892 2638 consensus_queue.cc:1048] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:11.835642 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f709db684b6c49cbaeef3bf8d76f5828 has no consensus state
I20251024 19:48:11.847287 1767 tablet_bootstrap.cc:492] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:11.847473 1767 ts_tablet_manager.cc:1403] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.003s
I20251024 19:48:11.850529 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "452f77091adc4cf09de2231f2f6d845b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:11.852017 1368 raft_consensus.cc:3060] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Advancing to term 3
I20251024 19:48:11.852036 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdfa5eea3b674a3389a156d1c72204df" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:11.825182 1305 leader_election.cc:304] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
W20251024 19:48:11.853268 1447 leader_election.cc:343] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.853957 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b88a6a5237d146f5b6f082ca3af25b2b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:11.854765 1447 leader_election.cc:343] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.859206 1768 tablet_bootstrap.cc:492] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:11.859432 1768 ts_tablet_manager.cc:1403] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.272s user 0.002s sys 0.014s
I20251024 19:48:11.860455 1768 raft_consensus.cc:359] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.866889 2569 consensus_queue.cc:1048] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:11.867048 1768 raft_consensus.cc:385] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.867131 1768 raft_consensus.cc:740] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.868242 1768 consensus_queue.cc:260] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:11.874845 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_823526f1c3164a3ba70c44dd8388fb9c" } modify_external_catalogs: true
I20251024 19:48:11.874902 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_587f4b6c35e94fbca64f79a03aecf9c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:11.875085 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_823526f1c3164a3ba70c44dd8388fb9c" } modify_external_catalogs: true
W20251024 19:48:11.875520 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_587f4b6c35e94fbca64f79a03aecf9c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.876365 1767 raft_consensus.cc:359] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.878527 1188 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 7503b22d9e12481b9af96385b9d76e50 (table default.Table_e5c532eb7fc241039c7e06e4757ea011 [id=9065c680f4da42b5a6e5ca335c8e2f3b]). Expected version 3 got 0
I20251024 19:48:11.878732 1767 raft_consensus.cc:385] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:11.879133 1767 raft_consensus.cc:740] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:11.879645 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c8ab8e29b5a94252a5395f467f5e130f" } new_table_name: "default.Table_aa0fb2d0d5f7498a98658f906b93b741" modify_external_catalogs: true
I20251024 19:48:11.882124 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6501382adb09425e8fa508d81fe78a24" } modify_external_catalogs: true
I20251024 19:48:11.882540 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6501382adb09425e8fa508d81fe78a24" } modify_external_catalogs: true
I20251024 19:48:11.885438 1768 ts_tablet_manager.cc:1434] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20251024 19:48:11.889781 1183 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 03cdbd1b62b741249ad78562a7edfd22 (table default.Table_42c3a4a03db14d0684721ce6af6960c2 [id=ca742ae585e0406abf43193a23aa9731]). Expected version 2 got 0
I20251024 19:48:11.891880 1369 raft_consensus.cc:2468] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:11.894707 2561 raft_consensus.cc:515] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.895416 2561 leader_election.cc:290] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.897491 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "701c93a600964c20a275f5942a2ff863" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:11.898756 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 85c7134bb2f046319868d41df161572c (DEFAULT_TABLE table=default.Table_c3cc8727161245ab88c9c67c4b45546f [id=af343accd846404bb2d49c401f646932]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.899276 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 46a740e6ac1b49138a5a3523961107f1 (DEFAULT_TABLE table=default.Table_4be8c37e3871442480b6f9c25bb75797 [id=198216eca4fd430aa72fbffa70da3bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.899874 1482 tablet_service.cc:1505] Processing CreateTablet for tablet b19cce078dc7426aacbb60f45379e506 (DEFAULT_TABLE table=default.Table_1ea2bdfecf104bf18dacacebb8b5dcc1 [id=7a7b26f8a6dd4f7ba3f4959fe1ea63fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.900908 1483 tablet_service.cc:1505] Processing CreateTablet for tablet c2ab00d07d27457882a9f35cc58eb5aa (DEFAULT_TABLE table=default.Table_6d8df15af842462fb931f649dc0dd806 [id=a0f6efea5806456f968378562ffbfdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.901448 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 16a8c772ac404ae2b2f86d948a0fcd2f (DEFAULT_TABLE table=default.Table_7d5f945cec1f41f2abd299341bb4fff2 [id=6a44b15fbace459099418ce3f8117013]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.905038 1349 tablet_service.cc:1505] Processing CreateTablet for tablet c2ab00d07d27457882a9f35cc58eb5aa (DEFAULT_TABLE table=default.Table_6d8df15af842462fb931f649dc0dd806 [id=a0f6efea5806456f968378562ffbfdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.905156 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 85c7134bb2f046319868d41df161572c (DEFAULT_TABLE table=default.Table_c3cc8727161245ab88c9c67c4b45546f [id=af343accd846404bb2d49c401f646932]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.905587 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 16a8c772ac404ae2b2f86d948a0fcd2f (DEFAULT_TABLE table=default.Table_7d5f945cec1f41f2abd299341bb4fff2 [id=6a44b15fbace459099418ce3f8117013]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.905689 1343 tablet_service.cc:1505] Processing CreateTablet for tablet b19cce078dc7426aacbb60f45379e506 (DEFAULT_TABLE table=default.Table_1ea2bdfecf104bf18dacacebb8b5dcc1 [id=7a7b26f8a6dd4f7ba3f4959fe1ea63fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.906111 1348 tablet_service.cc:1505] Processing CreateTablet for tablet e765b2ec82294d6e8c417e7674631d31 (DEFAULT_TABLE table=default.Table_ab36bd29826c4100968322ae6f68f6ac [id=3a69336a3373480ab824ed16415f004d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.906193 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 687ae2d88b894d6393788527e7e32ae4 (DEFAULT_TABLE table=default.Table_77837a284ba748f6852b42cc36890488 [id=c4087e3eaacf4c20ba2cf31f86a0a512]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.906698 2613 raft_consensus.cc:515] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.907037 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 4289d6b43f32464696118e5840bad849 (DEFAULT_TABLE table=default.Table_bb7717f8140d4bc6a90455b84c9906ea [id=3b138f885ea64d419a6897919d3b573e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.907318 2613 leader_election.cc:290] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.907653 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 2e6ab4e8ffb84bf0961af481f0d4b180 (DEFAULT_TABLE table=default.Table_e72407cfebf14ad2be5ee53e418fda62 [id=9ebff076ee594927a0ae62f9d4222ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.907964 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6ab4e8ffb84bf0961af481f0d4b180. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.908623 1183 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet d692e1988d13460fb1d54e930e17fc9d (table default.Table_b09229e0e1254adeacfed952befaa65a [id=67a0be12c85648bdb4515a096db21d3b]). Expected version 1 got 0
I20251024 19:48:11.909066 1368 raft_consensus.cc:2468] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 3.
I20251024 19:48:11.909461 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4289d6b43f32464696118e5840bad849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.909680 1774 tablet_replica.cc:333] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:11.910110 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e765b2ec82294d6e8c417e7674631d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.910521 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85c7134bb2f046319868d41df161572c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.910924 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b19cce078dc7426aacbb60f45379e506. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.911238 2656 raft_consensus.cc:2749] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:11.911267 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 7ec3afdbd80848e19fb0c4cd5b0eb0e7 (DEFAULT_TABLE table=default.Table_a5046911ec4642bd811676fbd8782d9a [id=08a3bfb8b8654b6b8c7646e35d8f9418]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.891523 1650 raft_consensus.cc:2468] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 3.
I20251024 19:48:11.912421 1616 tablet_service.cc:1505] Processing CreateTablet for tablet e765b2ec82294d6e8c417e7674631d31 (DEFAULT_TABLE table=default.Table_ab36bd29826c4100968322ae6f68f6ac [id=3a69336a3373480ab824ed16415f004d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.902624 1617 tablet_service.cc:1505] Processing CreateTablet for tablet c2ab00d07d27457882a9f35cc58eb5aa (DEFAULT_TABLE table=default.Table_6d8df15af842462fb931f649dc0dd806 [id=a0f6efea5806456f968378562ffbfdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.902624 1627 tablet_service.cc:1505] Processing CreateTablet for tablet 16a8c772ac404ae2b2f86d948a0fcd2f (DEFAULT_TABLE table=default.Table_7d5f945cec1f41f2abd299341bb4fff2 [id=6a44b15fbace459099418ce3f8117013]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.912868 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e765b2ec82294d6e8c417e7674631d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.913055 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a8c772ac404ae2b2f86d948a0fcd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.915172 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 2e6ab4e8ffb84bf0961af481f0d4b180 (DEFAULT_TABLE table=default.Table_e72407cfebf14ad2be5ee53e418fda62 [id=9ebff076ee594927a0ae62f9d4222ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.915503 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6ab4e8ffb84bf0961af481f0d4b180. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.918066 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2ab00d07d27457882a9f35cc58eb5aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.918123 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a8c772ac404ae2b2f86d948a0fcd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.918694 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2ab00d07d27457882a9f35cc58eb5aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.919989 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 687ae2d88b894d6393788527e7e32ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.920172 1767 consensus_queue.cc:260] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.903046 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 46a740e6ac1b49138a5a3523961107f1 (DEFAULT_TABLE table=default.Table_4be8c37e3871442480b6f9c25bb75797 [id=198216eca4fd430aa72fbffa70da3bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.921698 1767 ts_tablet_manager.cc:1434] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.074s user 0.001s sys 0.001s
I20251024 19:48:11.921880 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46a740e6ac1b49138a5a3523961107f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.922011 1767 tablet_bootstrap.cc:492] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:11.903553 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 687ae2d88b894d6393788527e7e32ae4 (DEFAULT_TABLE table=default.Table_77837a284ba748f6852b42cc36890488 [id=c4087e3eaacf4c20ba2cf31f86a0a512]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.903689 2569 raft_consensus.cc:493] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.922993 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 687ae2d88b894d6393788527e7e32ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.922897 2569 raft_consensus.cc:515] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.923525 2569 leader_election.cc:290] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.904223 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "701c93a600964c20a275f5942a2ff863" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:11.936642 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "635a132735704cfda69bf7c8deaea81d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:11.934132 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 7ec3afdbd80848e19fb0c4cd5b0eb0e7 (DEFAULT_TABLE table=default.Table_a5046911ec4642bd811676fbd8782d9a [id=08a3bfb8b8654b6b8c7646e35d8f9418]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.938082 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ec3afdbd80848e19fb0c4cd5b0eb0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.938457 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "635a132735704cfda69bf7c8deaea81d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:11.938741 1510 raft_consensus.cc:2468] T 635a132735704cfda69bf7c8deaea81d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:11.939081 2569 consensus_queue.cc:1048] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:11.934310 1647 raft_consensus.cc:3060] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.939616 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet d76410afd7ba4c73836f4791ef739056 (table default.Table_34bbddd692a44f46a4925c957f4e8848 [id=6f9b0ed7552c4556afd0f59316831a91]) successfully deleted
I20251024 19:48:11.939656 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 300952047a324a25a7c687eff20d6e56 (DEFAULT_TABLE table=default.Table_6811639fccc645fd9ccd5bfe038fbadd [id=003f20c882314503a8334a05bc68b955]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.940047 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 300952047a324a25a7c687eff20d6e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.935684 1622 tablet_service.cc:1505] Processing CreateTablet for tablet b19cce078dc7426aacbb60f45379e506 (DEFAULT_TABLE table=default.Table_1ea2bdfecf104bf18dacacebb8b5dcc1 [id=7a7b26f8a6dd4f7ba3f4959fe1ea63fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.940726 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b19cce078dc7426aacbb60f45379e506. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.942353 1647 raft_consensus.cc:2468] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:11.936689 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 85c7134bb2f046319868d41df161572c (DEFAULT_TABLE table=default.Table_c3cc8727161245ab88c9c67c4b45546f [id=af343accd846404bb2d49c401f646932]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.923676 1767 tablet_bootstrap.cc:654] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:11.945021 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85c7134bb2f046319868d41df161572c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.945892 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 10d29f40babf4cab98a702d4a4235790 (DEFAULT_TABLE table=default.Table_81ca5ba9d12c41cd9067ca6fef94e0b0 [id=99f3e01e305941718084dc2baab1e03a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.946154 1586 leader_election.cc:304] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:11.946262 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d29f40babf4cab98a702d4a4235790. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:11.946574 1586 leader_election.cc:343] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.947065 2640 raft_consensus.cc:2804] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:11.947162 2640 raft_consensus.cc:493] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:11.947304 2640 raft_consensus.cc:3060] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.949740 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 27b32126d04e480f8db7b2b20aba5873 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.950511 2640 raft_consensus.cc:515] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.951128 2640 leader_election.cc:290] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.951287 2640 consensus_queue.cc:1048] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:11.952564 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b1bf5228ae5f485abeb61ea7d512b30d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:11.952952 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85c7134bb2f046319868d41df161572c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:11.953197 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1bf5228ae5f485abeb61ea7d512b30d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.953634 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46a740e6ac1b49138a5a3523961107f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.954231 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2ab00d07d27457882a9f35cc58eb5aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.954645 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a8c772ac404ae2b2f86d948a0fcd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.954589 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "635a132735704cfda69bf7c8deaea81d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:11.954797 1444 leader_election.cc:304] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:11.955209 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b19cce078dc7426aacbb60f45379e506. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.955256 2567 raft_consensus.cc:2804] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 3 FOLLOWER]: Leader election won for term 3
I20251024 19:48:11.955399 2567 raft_consensus.cc:697] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 3 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:11.955647 2567 consensus_queue.cc:237] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.957082 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "635a132735704cfda69bf7c8deaea81d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:11.957264 1510 raft_consensus.cc:3060] T 635a132735704cfda69bf7c8deaea81d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:11.960045 1510 raft_consensus.cc:2468] T 635a132735704cfda69bf7c8deaea81d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
W20251024 19:48:11.907279 1447 leader_election.cc:343] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:11.964308 1447 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 57041.6us
I20251024 19:48:11.964857 1488 tablet_service.cc:1505] Processing CreateTablet for tablet e765b2ec82294d6e8c417e7674631d31 (DEFAULT_TABLE table=default.Table_ab36bd29826c4100968322ae6f68f6ac [id=3a69336a3373480ab824ed16415f004d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.965282 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e765b2ec82294d6e8c417e7674631d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.965795 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_96fd924db35c4970944db55c46bace57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:11.966353 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96fd924db35c4970944db55c46bace57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:11.969728 1444 leader_election.cc:304] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
W20251024 19:48:11.970450 1444 consensus_peers.cc:597] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:11.907624 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 687ae2d88b894d6393788527e7e32ae4 (DEFAULT_TABLE table=default.Table_77837a284ba748f6852b42cc36890488 [id=c4087e3eaacf4c20ba2cf31f86a0a512]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.973892 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 687ae2d88b894d6393788527e7e32ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.977089 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 7ec3afdbd80848e19fb0c4cd5b0eb0e7 (DEFAULT_TABLE table=default.Table_a5046911ec4642bd811676fbd8782d9a [id=08a3bfb8b8654b6b8c7646e35d8f9418]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.977561 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ec3afdbd80848e19fb0c4cd5b0eb0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.978688 2656 raft_consensus.cc:493] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:11.978821 2656 raft_consensus.cc:515] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.914687 2615 raft_consensus.cc:515] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.979369 2656 leader_election.cc:290] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:11.979622 2615 leader_election.cc:290] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:11.979862 2615 raft_consensus.cc:2804] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:11.980039 2615 raft_consensus.cc:697] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:11.980317 2615 consensus_queue.cc:237] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:11.980450 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdfa5eea3b674a3389a156d1c72204df" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:11.980320 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 2e6ab4e8ffb84bf0961af481f0d4b180 (DEFAULT_TABLE table=default.Table_e72407cfebf14ad2be5ee53e418fda62 [id=9ebff076ee594927a0ae62f9d4222ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.981145 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6ab4e8ffb84bf0961af481f0d4b180. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.981217 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet cef80e4d14bb4e719bceebf43041d079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.981870 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 4289d6b43f32464696118e5840bad849 (DEFAULT_TABLE table=default.Table_bb7717f8140d4bc6a90455b84c9906ea [id=3b138f885ea64d419a6897919d3b573e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.982189 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 701c93a600964c20a275f5942a2ff863 with delete_type TABLET_DATA_DELETED (replaced by tablet 7da1601a87524ef1a8b5069ba6823843) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.982297 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4289d6b43f32464696118e5840bad849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.982713 1627 tablet_service.cc:1552] Processing DeleteTablet for tablet b88a6a5237d146f5b6f082ca3af25b2b with delete_type TABLET_DATA_DELETED (replaced by tablet 639836c656f44a819036f4d00e055bd9) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:11.982580 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 46a740e6ac1b49138a5a3523961107f1 (DEFAULT_TABLE table=default.Table_4be8c37e3871442480b6f9c25bb75797 [id=198216eca4fd430aa72fbffa70da3bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.986069 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 300952047a324a25a7c687eff20d6e56 (DEFAULT_TABLE table=default.Table_6811639fccc645fd9ccd5bfe038fbadd [id=003f20c882314503a8334a05bc68b955]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.986428 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 300952047a324a25a7c687eff20d6e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.988411 1586 leader_election.cc:304] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:11.988720 1586 leader_election.cc:343] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:11.989259 1338 tablet_service.cc:1505] Processing CreateTablet for tablet a64e3591bc544283b409a0666fbcd865 (DEFAULT_TABLE table=default.Table_575ba6a1b4c64e77aadef06e3084a52a [id=b54040c991354937a9ba3afdd9a75470]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.989720 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a64e3591bc544283b409a0666fbcd865. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.990064 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46a740e6ac1b49138a5a3523961107f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.995098 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 4289d6b43f32464696118e5840bad849 (DEFAULT_TABLE table=default.Table_bb7717f8140d4bc6a90455b84c9906ea [id=3b138f885ea64d419a6897919d3b573e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.995528 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4289d6b43f32464696118e5840bad849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:11.996770 2640 raft_consensus.cc:2804] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:11.996907 2640 raft_consensus.cc:697] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:11.997231 2640 consensus_queue.cc:237] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:11.997314 1305 leader_election.cc:343] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 7352d84226db467dad31ae5941842b80
I20251024 19:48:11.996704 1627 tablet_service.cc:1505] Processing CreateTablet for tablet 10d29f40babf4cab98a702d4a4235790 (DEFAULT_TABLE table=default.Table_81ca5ba9d12c41cd9067ca6fef94e0b0 [id=99f3e01e305941718084dc2baab1e03a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:11.998797 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d29f40babf4cab98a702d4a4235790. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.001791 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ec3afdbd80848e19fb0c4cd5b0eb0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.009737 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_aa1ad4ed1e214bb890edff3ebe855022"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.010340 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa1ad4ed1e214bb890edff3ebe855022 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.011806 1339 tablet_service.cc:1505] Processing CreateTablet for tablet da4315a1ac6948938e76da54924ce248 (DEFAULT_TABLE table=default.Table_86c533ff0d26405bb7a1c203c39d18a0 [id=9f790b431caf410db28053df4923702a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.012267 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da4315a1ac6948938e76da54924ce248. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.012894 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 300952047a324a25a7c687eff20d6e56 (DEFAULT_TABLE table=default.Table_6811639fccc645fd9ccd5bfe038fbadd [id=003f20c882314503a8334a05bc68b955]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.013406 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 300952047a324a25a7c687eff20d6e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.016623 1620 tablet_service.cc:1505] Processing CreateTablet for tablet a64e3591bc544283b409a0666fbcd865 (DEFAULT_TABLE table=default.Table_575ba6a1b4c64e77aadef06e3084a52a [id=b54040c991354937a9ba3afdd9a75470]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.017097 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a64e3591bc544283b409a0666fbcd865. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.024348 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b88a6a5237d146f5b6f082ca3af25b2b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:12.024575 1650 raft_consensus.cc:3060] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:12.027287 1650 raft_consensus.cc:2468] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:12.027333 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet ce8c179e98944df0b510a0e8daf3920f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:12.028169 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7352d84226db467dad31ae5941842b80" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:12.030546 1775 ts_tablet_manager.cc:1929] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:12.030723 1775 log.cc:1199] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/e3b4e85f741849b2a6a3e8574396ff88
I20251024 19:48:12.031229 1775 ts_tablet_manager.cc:1950] T e3b4e85f741849b2a6a3e8574396ff88 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:12.032303 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 480) took 6050 ms. Trace:
I20251024 19:48:12.032896 1308 leader_election.cc:304] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:12.033944 1444 leader_election.cc:304] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:12.034240 2656 raft_consensus.cc:2749] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:12.034657 2561 raft_consensus.cc:2804] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:12.034821 2561 raft_consensus.cc:697] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:12.035113 2561 consensus_queue.cc:237] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.037993 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet eee675874516489f9e8dac41f70ac4e4
I20251024 19:48:12.038849 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 844638a6b6944d0394df3e78a8ea51d6
I20251024 19:48:12.038775 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3253b4bae4f14474b0c319b84e6b3130"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:12.039018 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 64d8da45c6364f1e9ddc79307904587c
W20251024 19:48:12.039499 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3253b4bae4f14474b0c319b84e6b3130 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.039700 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_babcfed7fd5b4b6ebf17c81871843326"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:12.040357 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdfa5eea3b674a3389a156d1c72204df" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:12.040594 1647 raft_consensus.cc:3060] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:12.040617 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_babcfed7fd5b4b6ebf17c81871843326 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.040534 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a478a67f294e4fdcba8d2f2b421d26fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.041136 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a478a67f294e4fdcba8d2f2b421d26fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:12.041874 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7172a93a5bec4fa1b63282ba63beec0c consensus state has no leader
I20251024 19:48:12.042245 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ab36bd29826c4100968322ae6f68f6ac" } new_table_name: "default.Table_659e36705f5646afb4c4e0ceb75b7278" modify_external_catalogs: true
I20251024 19:48:12.042994 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_16e10ff29f654c72802003f0c316b89f" } new_table_name: "default.Table_e81edbfed6c14c7cb43ba38ef86e78ee" modify_external_catalogs: true
I20251024 19:48:12.043574 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 10d29f40babf4cab98a702d4a4235790 (DEFAULT_TABLE table=default.Table_81ca5ba9d12c41cd9067ca6fef94e0b0 [id=99f3e01e305941718084dc2baab1e03a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:12.043793 1447 leader_election.cc:343] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:12.044014 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d29f40babf4cab98a702d4a4235790. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.045382 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b88a6a5237d146f5b6f082ca3af25b2b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:12.045688 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 64d8da45c6364f1e9ddc79307904587c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:12.046049 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 64d8da45c6364f1e9ddc79307904587c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:12.046247 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet 844638a6b6944d0394df3e78a8ea51d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:12.046474 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet eee675874516489f9e8dac41f70ac4e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:12.050089 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 844638a6b6944d0394df3e78a8ea51d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:12.050745 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 64d8da45c6364f1e9ddc79307904587c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:12.044020 1447 consensus_peers.cc:597] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:12.051466 1628 tablet_service.cc:1505] Processing CreateTablet for tablet da4315a1ac6948938e76da54924ce248 (DEFAULT_TABLE table=default.Table_86c533ff0d26405bb7a1c203c39d18a0 [id=9f790b431caf410db28053df4923702a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.051640 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 5ccaab239a01480ebea7dd81878b91f7 (DEFAULT_TABLE table=default.Table_0f16fa751e6244afbe2f1683e608bc81 [id=8272a9bf02544e7b84bb4e56fd41f567]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.051995 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ccaab239a01480ebea7dd81878b91f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.052168 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da4315a1ac6948938e76da54924ce248. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:12.052014 1447 leader_election.cc:343] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:12.053769 1483 tablet_service.cc:1505] Processing CreateTablet for tablet da4315a1ac6948938e76da54924ce248 (DEFAULT_TABLE table=default.Table_86c533ff0d26405bb7a1c203c39d18a0 [id=9f790b431caf410db28053df4923702a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.054188 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da4315a1ac6948938e76da54924ce248. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.058275 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet eee675874516489f9e8dac41f70ac4e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:12.059481 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 844638a6b6944d0394df3e78a8ea51d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:12.059896 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet eee675874516489f9e8dac41f70ac4e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:12.061884 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 55897.1us
I20251024 19:48:12.063200 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 5ccaab239a01480ebea7dd81878b91f7 (DEFAULT_TABLE table=default.Table_0f16fa751e6244afbe2f1683e608bc81 [id=8272a9bf02544e7b84bb4e56fd41f567]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.063716 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ccaab239a01480ebea7dd81878b91f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.065981 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 5ccaab239a01480ebea7dd81878b91f7 (DEFAULT_TABLE table=default.Table_0f16fa751e6244afbe2f1683e608bc81 [id=8272a9bf02544e7b84bb4e56fd41f567]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.068249 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ccaab239a01480ebea7dd81878b91f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.068380 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 3a99e9b3fe044c7eb14fa9bffa903873 (DEFAULT_TABLE table=default.Table_9bfa6cbfde5148d78028b23509294e53 [id=831b5fb847ab42a8b60d830226f599ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.068790 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a99e9b3fe044c7eb14fa9bffa903873. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.068936 1768 tablet_bootstrap.cc:492] T 187925082cdd4923a6e9148bf9f270fd P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:12.067912 1767 tablet_bootstrap.cc:492] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:12.069260 1767 ts_tablet_manager.cc:1403] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.147s user 0.004s sys 0.000s
I20251024 19:48:12.069602 2656 raft_consensus.cc:493] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.069746 2656 raft_consensus.cc:515] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.070158 1767 raft_consensus.cc:359] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.070307 2656 leader_election.cc:290] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:12.070343 1767 raft_consensus.cc:385] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.070412 1767 raft_consensus.cc:740] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:12.070698 1768 tablet_bootstrap.cc:654] T 187925082cdd4923a6e9148bf9f270fd P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:12.070650 1767 consensus_queue.cc:260] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.071247 1767 ts_tablet_manager.cc:1434] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:12.071359 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73ffe342f541447d8519ee24bc98a0f3" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:12.071651 1510 raft_consensus.cc:2393] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:12.071666 1767 tablet_bootstrap.cc:492] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:12.073441 1767 tablet_bootstrap.cc:654] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:12.068347 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 3a99e9b3fe044c7eb14fa9bffa903873 (DEFAULT_TABLE table=default.Table_9bfa6cbfde5148d78028b23509294e53 [id=831b5fb847ab42a8b60d830226f599ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.074949 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a99e9b3fe044c7eb14fa9bffa903873. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.075769 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet dc14b0cdb7f342ac956a7857ab3f35d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:12.076514 1305 leader_election.cc:343] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 73ffe342f541447d8519ee24bc98a0f3
I20251024 19:48:12.076637 1305 leader_election.cc:304] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:12.076960 2656 raft_consensus.cc:3060] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:12.079806 2656 raft_consensus.cc:2749] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:12.084116 1647 raft_consensus.cc:2468] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:12.084635 1444 leader_election.cc:304] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:12.085559 2561 raft_consensus.cc:2804] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:12.087208 2656 raft_consensus.cc:493] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.087400 2656 raft_consensus.cc:515] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.088231 2656 leader_election.cc:290] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:12.088836 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2037fcca87b94b5fb31639b4d07307b0" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:12.089213 1510 raft_consensus.cc:2393] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:12.090891 2561 raft_consensus.cc:697] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:12.092490 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 3a99e9b3fe044c7eb14fa9bffa903873 (DEFAULT_TABLE table=default.Table_9bfa6cbfde5148d78028b23509294e53 [id=831b5fb847ab42a8b60d830226f599ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.093542 1774 raft_consensus.cc:2243] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:12.093978 1774 pending_rounds.cc:70] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:12.094038 1774 pending_rounds.cc:77] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429337640431616 op_type: NO_OP noop_request { }
I20251024 19:48:12.094146 1774 raft_consensus.cc:2889] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:12.094239 1774 raft_consensus.cc:2272] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:12.094606 1177 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.254:36505, user_credentials={real_user=slave}} blocked reactor thread for 59395.2us
I20251024 19:48:12.097246 2561 consensus_queue.cc:237] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.097638 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 373aa543e1284fbc945ca802d9eddf59
I20251024 19:48:12.099808 1186 catalog_manager.cc:5649] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:12.100210 1186 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 635a132735704cfda69bf7c8deaea81d (table default.Table_933a6697624149369ab3ec28462b466a [id=93a61862707b4a7ba8a0ff349704dc2b]). Expected version 1 got 0
I20251024 19:48:12.100337 1767 tablet_bootstrap.cc:492] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:12.100244 1490 tablet_service.cc:1505] Processing CreateTablet for tablet a64e3591bc544283b409a0666fbcd865 (DEFAULT_TABLE table=default.Table_575ba6a1b4c64e77aadef06e3084a52a [id=b54040c991354937a9ba3afdd9a75470]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.100690 1767 ts_tablet_manager.cc:1403] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20251024 19:48:12.101164 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a64e3591bc544283b409a0666fbcd865. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.102391 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 373aa543e1284fbc945ca802d9eddf59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:12.102991 1767 raft_consensus.cc:359] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.103271 1767 raft_consensus.cc:385] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.103410 1767 raft_consensus.cc:740] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:12.103950 1767 consensus_queue.cc:260] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.104751 1767 ts_tablet_manager.cc:1434] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:12.106374 2569 raft_consensus.cc:493] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.106578 2569 raft_consensus.cc:515] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.107276 2569 leader_election.cc:290] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:12.108330 1767 tablet_bootstrap.cc:492] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:12.108932 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce8c179e98944df0b510a0e8daf3920f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:12.032548 1775 rpcz_store.cc:276] 1024 19:48:05.982118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.010503 (+ 28385us) service_pool.cc:225] Handling call
1024 19:48:11.794621 (+5784118us) ts_tablet_manager.cc:1151] Deleting tablet e3b4e85f741849b2a6a3e8574396ff88
1024 19:48:11.794630 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:11.990770 (+196140us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.030428 (+ 39658us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.032247 (+ 1819us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5758957}
I20251024 19:48:12.110213 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet e3b4e85f741849b2a6a3e8574396ff88 (table default.Table_2a9850f5d5dc499f87e87bdb0efcc93a [id=1da47e6010f548d1b3d8c538150a8a8e]) successfully deleted
I20251024 19:48:12.110862 1775 tablet_replica.cc:333] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:12.111196 1775 raft_consensus.cc:2243] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:12.111474 1775 pending_rounds.cc:70] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e: Trying to abort 1 pending ops.
I20251024 19:48:12.111615 1775 pending_rounds.cc:77] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429342251991040 op_type: NO_OP noop_request { }
I20251024 19:48:12.111838 1775 raft_consensus.cc:2889] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:12.112016 1775 raft_consensus.cc:2272] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:12.112725 1775 ts_tablet_manager.cc:1916] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:12.113047 1586 leader_election.cc:343] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:12.113962 1774 ts_tablet_manager.cc:1916] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:12.117985 2747 raft_consensus.cc:493] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.118139 2747 raft_consensus.cc:515] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.118929 2747 leader_election.cc:290] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:12.119886 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b32126d04e480f8db7b2b20aba5873" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:12.120296 1767 tablet_bootstrap.cc:654] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:12.120759 1586 leader_election.cc:343] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:12.128121 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a99e9b3fe044c7eb14fa9bffa903873. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.131783 1767 tablet_bootstrap.cc:492] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:12.131858 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 373aa543e1284fbc945ca802d9eddf59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:12.132040 1767 ts_tablet_manager.cc:1403] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.004s
I20251024 19:48:12.132520 1768 tablet_bootstrap.cc:492] T 187925082cdd4923a6e9148bf9f270fd P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:12.132658 1768 ts_tablet_manager.cc:1403] T 187925082cdd4923a6e9148bf9f270fd P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20251024 19:48:12.133335 1767 raft_consensus.cc:359] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.133594 1767 raft_consensus.cc:385] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.133701 1767 raft_consensus.cc:740] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:12.134004 1767 consensus_queue.cc:260] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.134570 1767 ts_tablet_manager.cc:1434] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:12.135075 1767 tablet_bootstrap.cc:492] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:12.138038 1767 tablet_bootstrap.cc:654] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:12.119738 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b32126d04e480f8db7b2b20aba5873" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:12.155265 1510 raft_consensus.cc:2468] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:12.155822 1586 leader_election.cc:304] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:12.156105 2569 raft_consensus.cc:2804] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:12.156225 2569 raft_consensus.cc:493] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:12.156373 2569 raft_consensus.cc:3060] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:12.160542 2569 raft_consensus.cc:515] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.162763 1766 tablet_bootstrap.cc:492] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:12.163069 1766 ts_tablet_manager.cc:1403] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.395s user 0.015s sys 0.011s
I20251024 19:48:12.163143 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce8c179e98944df0b510a0e8daf3920f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:12.164367 1766 raft_consensus.cc:359] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.166543 1766 raft_consensus.cc:385] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.167871 1766 raft_consensus.cc:740] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
W20251024 19:48:12.163717 1586 leader_election.cc:343] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:12.176182 1586 leader_election.cc:304] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:12.169834 1766 consensus_queue.cc:260] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.176492 2638 raft_consensus.cc:2749] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:12.166345 1774 ts_tablet_manager.cc:1929] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:12.178452 1774 log.cc:1199] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/5fd7726c532145c6b44ea9a37544510d
I20251024 19:48:12.179343 1766 ts_tablet_manager.cc:1434] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20251024 19:48:12.179754 1766 tablet_bootstrap.cc:492] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:12.183437 1768 raft_consensus.cc:359] T 187925082cdd4923a6e9148bf9f270fd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.183713 1768 raft_consensus.cc:385] T 187925082cdd4923a6e9148bf9f270fd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.183797 1768 raft_consensus.cc:740] T 187925082cdd4923a6e9148bf9f270fd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:12.184069 1768 consensus_queue.cc:260] T 187925082cdd4923a6e9148bf9f270fd P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.184937 2640 raft_consensus.cc:493] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.185189 2640 raft_consensus.cc:515] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.186371 2640 leader_election.cc:290] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:12.171463 1776 ts_tablet_manager.cc:1929] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:12.187619 1776 log.cc:1199] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/b79801594e95434cbb8fe7b75b14602e
I20251024 19:48:12.189886 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "187925082cdd4923a6e9148bf9f270fd" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:12.190135 1364 raft_consensus.cc:2468] T 187925082cdd4923a6e9148bf9f270fd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:12.190553 1774 ts_tablet_manager.cc:1950] T 5fd7726c532145c6b44ea9a37544510d P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:12.190171 2569 leader_election.cc:290] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:12.190688 1586 leader_election.cc:304] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:12.191627 2640 raft_consensus.cc:2804] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:12.191818 2640 raft_consensus.cc:493] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:12.191973 2640 raft_consensus.cc:3060] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:12.196214 2640 raft_consensus.cc:515] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.206211 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b32126d04e480f8db7b2b20aba5873" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:12.207607 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "187925082cdd4923a6e9148bf9f270fd" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:12.190948 1776 ts_tablet_manager.cc:1950] T b79801594e95434cbb8fe7b75b14602e P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:12.212306 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "187925082cdd4923a6e9148bf9f270fd" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:12.213106 1511 raft_consensus.cc:2468] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:12.214987 1766 tablet_bootstrap.cc:654] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:12.215339 1510 raft_consensus.cc:3060] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:12.219345 1774 tablet_replica.cc:333] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:12.219693 1774 raft_consensus.cc:2243] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:12.220010 1774 raft_consensus.cc:2272] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:12.231958 1510 raft_consensus.cc:2468] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:12.241843 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f57b3f3656d347a3aee8146fd991ca1e" } new_table_name: "default.Table_2f890170e9354b4f871df35ba034baa2" modify_external_catalogs: true
I20251024 19:48:12.244029 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_867f1b66094349fa8d81af1e89bebb06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.244722 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_867f1b66094349fa8d81af1e89bebb06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.246266 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 57fed826b4b64edba48c5fffaef1e031
I20251024 19:48:12.247552 1189 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 7503b22d9e12481b9af96385b9d76e50 (table default.Table_e5c532eb7fc241039c7e06e4757ea011 [id=9065c680f4da42b5a6e5ca335c8e2f3b]). Expected version 3 got 0
I20251024 19:48:12.247777 1189 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet a3259823b622461fa823841925b3d203 (table default.Table_ea0d3589c13a4ebeb950ddc737a9948b [id=350b37bf42194fcfbb660bffe83e47c4]). Expected version 4 got 0
I20251024 19:48:12.248744 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_abc327da99a44b9d9f01f12c54b1ede2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:12.249171 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ecaba24cb11e46e980659c47d6cd0c49" } new_table_name: "default.Table_71699a436b8045e6a4f9abdcb32351da" modify_external_catalogs: true
W20251024 19:48:12.249267 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abc327da99a44b9d9f01f12c54b1ede2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:12.251848 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c96c11697514af6bf4e80a67aadf748 consensus state has no leader
I20251024 19:48:12.252251 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_35b42b8f59b042ba9161297fca0d9bd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.252823 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35b42b8f59b042ba9161297fca0d9bd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.252904 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 5fd7726c532145c6b44ea9a37544510d (table default.Table_999546c6f492460bb6dc96ab55b7d3ad [id=7dbbe53e70a547888a510250e0583e1d]) successfully deleted
W20251024 19:48:12.255556 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e765b2ec82294d6e8c417e7674631d31 consensus state has no leader
I20251024 19:48:12.256019 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ce9762cc2ae44e2dbf2404da0554444c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.256481 1447 consensus_peers.cc:597] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fd7726c532145c6b44ea9a37544510d. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:12.256713 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce9762cc2ae44e2dbf2404da0554444c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.260977 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_aa0fb2d0d5f7498a98658f906b93b741" } modify_external_catalogs: true
I20251024 19:48:12.261229 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_aa0fb2d0d5f7498a98658f906b93b741" } modify_external_catalogs: true
I20251024 19:48:12.262115 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 57fed826b4b64edba48c5fffaef1e031 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:12.264185 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0d796a82cf4b4e0f9cba35051c69293d" } modify_external_catalogs: true
I20251024 19:48:12.264892 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0d796a82cf4b4e0f9cba35051c69293d" } modify_external_catalogs: true
I20251024 19:48:12.269891 1774 ts_tablet_manager.cc:1916] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:12.271682 2756 raft_consensus.cc:493] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.272082 2756 raft_consensus.cc:515] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.273362 2756 leader_election.cc:290] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:12.274356 1444 leader_election.cc:343] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: fca73ab61f0c47df9186459caf2b9a32
I20251024 19:48:12.218693 1768 ts_tablet_manager.cc:1434] T 187925082cdd4923a6e9148bf9f270fd P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.086s user 0.002s sys 0.001s
W20251024 19:48:12.277742 1447 leader_election.cc:343] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: fca73ab61f0c47df9186459caf2b9a32
I20251024 19:48:12.277891 1447 leader_election.cc:304] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:12.278312 2756 raft_consensus.cc:2749] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:48:12.211048 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 373aa543e1284fbc945ca802d9eddf59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:12.225761 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "187925082cdd4923a6e9148bf9f270fd" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:12.233522 1586 leader_election.cc:304] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:12.279038 1364 raft_consensus.cc:3060] T 187925082cdd4923a6e9148bf9f270fd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:12.279121 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 45670.2us
I20251024 19:48:12.279354 2569 raft_consensus.cc:2804] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:12.279836 2569 raft_consensus.cc:697] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:12.280293 2569 consensus_queue.cc:237] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.197846 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b32126d04e480f8db7b2b20aba5873" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:12.282385 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5744ec49c5e349d79d7473d6ed4154c6" } modify_external_catalogs: true
W20251024 19:48:12.282462 1586 leader_election.cc:343] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:12.282621 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5744ec49c5e349d79d7473d6ed4154c6" } modify_external_catalogs: true
I20251024 19:48:12.260502 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 59c7dbe9ff2c4869994c4f43d5676c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:12.268818 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 57fed826b4b64edba48c5fffaef1e031 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:12.284307 1364 raft_consensus.cc:2468] T 187925082cdd4923a6e9148bf9f270fd P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:12.284788 1586 leader_election.cc:304] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
I20251024 19:48:12.285140 2747 raft_consensus.cc:2804] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:12.285265 2747 raft_consensus.cc:697] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:12.285495 2747 consensus_queue.cc:237] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.288249 1776 tablet_replica.cc:333] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:48:12.288946 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce5b2cfcc8294b3ea8b89334786006f8 consensus state has no leader
I20251024 19:48:12.290987 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e25a14d3a9674ad38cce670621e477df" } new_table_name: "default.Table_673109e8b8e744f99120d9f081b1dd9d" modify_external_catalogs: true
W20251024 19:48:12.291473 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 104176d304384d1b99b051ae5744eec8 has no consensus state
I20251024 19:48:12.292675 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7172a93a5bec4fa1b63282ba63beec0c
I20251024 19:48:12.295337 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet be8126459ef541f39bc133f27c7d0f59
I20251024 19:48:12.301528 1776 raft_consensus.cc:2243] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:12.301824 1776 raft_consensus.cc:2272] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:12.302642 1776 ts_tablet_manager.cc:1916] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:12.304147 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet be8126459ef541f39bc133f27c7d0f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:12.304436 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 7172a93a5bec4fa1b63282ba63beec0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:12.304666 2656 raft_consensus.cc:493] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.304831 2656 raft_consensus.cc:515] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.305397 2656 leader_election.cc:290] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:12.306414 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet da7d38b63708427982591da1014aca81
I20251024 19:48:12.306524 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 7172a93a5bec4fa1b63282ba63beec0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:12.306910 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet be8126459ef541f39bc133f27c7d0f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:12.307291 1308 leader_election.cc:343] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: ad81d80697984ff1834a7d795e0f4b9b
W20251024 19:48:12.311700 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d1b6eaa353c4284a2b114b42437272c consensus state has no leader
I20251024 19:48:12.312688 1768 tablet_bootstrap.cc:492] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:12.314033 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_08dbb5c3a0924b1491e302cef50518c4" } modify_external_catalogs: true
I20251024 19:48:12.314252 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_08dbb5c3a0924b1491e302cef50518c4" } modify_external_catalogs: true
W20251024 19:48:12.314679 1444 consensus_peers.cc:597] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7352d84226db467dad31ae5941842b80. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:12.315343 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet da7d38b63708427982591da1014aca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:12.319727 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet da7d38b63708427982591da1014aca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:12.322158 1369 raft_consensus.cc:1275] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Refusing update from remote peer 10a9536207e343f188e0ef709081d04e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251024 19:48:12.326709 1766 tablet_bootstrap.cc:492] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:12.327241 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet fb08e56db59b4db796c50db5bda93af9
I20251024 19:48:12.329195 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet fb08e56db59b4db796c50db5bda93af9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:12.329362 1766 ts_tablet_manager.cc:1403] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.150s user 0.008s sys 0.009s
I20251024 19:48:12.330533 1766 raft_consensus.cc:359] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.330770 1766 raft_consensus.cc:385] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.330876 1766 raft_consensus.cc:740] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:12.330880 2615 consensus_queue.cc:1048] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:12.331131 1766 consensus_queue.cc:260] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.331661 1766 ts_tablet_manager.cc:1434] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251024 19:48:12.331981 1766 tablet_bootstrap.cc:492] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:12.334767 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet fb08e56db59b4db796c50db5bda93af9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:12.338230 1775 ts_tablet_manager.cc:1929] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:12.340348 1766 tablet_bootstrap.cc:654] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:12.341948 1187 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 03cdbd1b62b741249ad78562a7edfd22 (table default.Table_42c3a4a03db14d0684721ce6af6960c2 [id=ca742ae585e0406abf43193a23aa9731]). Expected version 2 got 0
I20251024 19:48:12.342388 1187 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet a3259823b622461fa823841925b3d203 (table default.Table_ea0d3589c13a4ebeb950ddc737a9948b [id=350b37bf42194fcfbb660bffe83e47c4]). Expected version 4 got 0
I20251024 19:48:12.342178 1775 log.cc:1199] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/3754540ae21e4c65b5af3d0aa3f9189f
W20251024 19:48:12.347836 1444 consensus_peers.cc:597] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6661b9e9a174849b6093d09cd8c1251. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:12.348301 1185 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 187925082cdd4923a6e9148bf9f270fd (table default.Table_55f8c66bd97343438051a98af12d8c2c [id=33f5ac73298543d0b5af77a27e897b79]). Expected version 1 got 0
I20251024 19:48:12.348551 1185 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet a3259823b622461fa823841925b3d203 (table default.Table_ea0d3589c13a4ebeb950ddc737a9948b [id=350b37bf42194fcfbb660bffe83e47c4]). Expected version 4 got 0
I20251024 19:48:12.348742 1185 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 7503b22d9e12481b9af96385b9d76e50 (table default.Table_e5c532eb7fc241039c7e06e4757ea011 [id=9065c680f4da42b5a6e5ca335c8e2f3b]). Expected version 3 got 0
I20251024 19:48:12.351426 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 182314f0f3074150b139cb2476976e1a (DEFAULT_TABLE table=default.Table_587f4b6c35e94fbca64f79a03aecf9c1 [id=aa7fd0bcc8f141ceb3488072f873b025]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.354635 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 182314f0f3074150b139cb2476976e1a (DEFAULT_TABLE table=default.Table_587f4b6c35e94fbca64f79a03aecf9c1 [id=aa7fd0bcc8f141ceb3488072f873b025]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.356141 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 187925082cdd4923a6e9148bf9f270fd (table default.Table_55f8c66bd97343438051a98af12d8c2c [id=33f5ac73298543d0b5af77a27e897b79]). Expected version 1 got 0
I20251024 19:48:12.356146 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 1da699a314ce4984b7566145ffb74ef1 (DEFAULT_TABLE table=default.Table_475f0aa37b7b4cff876223e3d8a0514c [id=d4dbe7cda31e4901b0e1aed96d5670ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.358546 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 37163451258c436ab1f874dd0d66d393 (DEFAULT_TABLE table=default.Table_922f48f987904120a61a2eba3d270993 [id=1958f311b2f549888ecce9fb451136a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.358726 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 583d0ef19aad4a80a8da15f7abc481c2 (DEFAULT_TABLE table=default.Table_8488a13f677347b591b4efde0761451b [id=5d07be972d4743979b659603ab7c2da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.359478 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 9275741737ba44e69b18a9419a10d6a6 (DEFAULT_TABLE table=default.Table_092242c51bbb4695947456a8ea5513dc [id=65830805c2cc488fa128ce367145ec2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.360316 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 104176d304384d1b99b051ae5744eec8 (DEFAULT_TABLE table=default.Table_71699a436b8045e6a4f9abdcb32351da [id=1da6e53aaefb4170977ada79d336a35c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.361645 1333 tablet_service.cc:1505] Processing CreateTablet for tablet b22d207fbc234913b41a781c8836419e (DEFAULT_TABLE table=default.Table_135cfe439b8e4e44a4317938ff165eaf [id=97bd5f6c47e647d09c8e9c509a585891]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.362954 1345 tablet_service.cc:1505] Processing CreateTablet for tablet d0cbe0a3436a498a84c892ac156798ed (DEFAULT_TABLE table=default.Table_4dca2888019448ca97c68a8c6dd8396a [id=9da07f2aef964c9a838808d27f211636]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.295091 2640 leader_election.cc:290] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:12.326875 1586 consensus_peers.cc:597] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:12.370223 1768 tablet_bootstrap.cc:654] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:12.373312 1767 tablet_bootstrap.cc:492] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:12.373512 1767 ts_tablet_manager.cc:1403] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.238s user 0.040s sys 0.052s
I20251024 19:48:12.373457 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 37163451258c436ab1f874dd0d66d393 (DEFAULT_TABLE table=default.Table_922f48f987904120a61a2eba3d270993 [id=1958f311b2f549888ecce9fb451136a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.374002 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37163451258c436ab1f874dd0d66d393. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.374522 1767 raft_consensus.cc:359] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.374725 1767 raft_consensus.cc:385] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.374781 1767 raft_consensus.cc:740] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:12.375020 1767 consensus_queue.cc:260] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.375607 1767 ts_tablet_manager.cc:1434] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:12.375950 1767 tablet_bootstrap.cc:492] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:12.378535 1767 tablet_bootstrap.cc:654] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:12.378404 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 583d0ef19aad4a80a8da15f7abc481c2 (DEFAULT_TABLE table=default.Table_8488a13f677347b591b4efde0761451b [id=5d07be972d4743979b659603ab7c2da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.378923 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 583d0ef19aad4a80a8da15f7abc481c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.387650 2441 rpc_server.cc:307] RPC server started. Bound to: 127.0.252.254:36505
I20251024 19:48:12.388617 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 104176d304384d1b99b051ae5744eec8 (DEFAULT_TABLE table=default.Table_71699a436b8045e6a4f9abdcb32351da [id=1da6e53aaefb4170977ada79d336a35c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.390558 1491 tablet_service.cc:1505] Processing CreateTablet for tablet c694a372afc94f89972c972793c5e901 (DEFAULT_TABLE table=default.Table_43dbb020b49b44cfa627ef85edba938c [id=7d5fa7422f654970b1eee19e5f6d4360]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.390914 1486 tablet_service.cc:1505] Processing CreateTablet for tablet f709db684b6c49cbaeef3bf8d76f5828 (DEFAULT_TABLE table=default.Table_5bfbf5d7f5764b96a95bf3ac0e0aba69 [id=9a49120c5b654676a63cd3966eaf31a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.389293 1475 tablet_service.cc:1505] Processing CreateTablet for tablet b22d207fbc234913b41a781c8836419e (DEFAULT_TABLE table=default.Table_135cfe439b8e4e44a4317938ff165eaf [id=97bd5f6c47e647d09c8e9c509a585891]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.391371 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104176d304384d1b99b051ae5744eec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.391705 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f709db684b6c49cbaeef3bf8d76f5828. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.388898 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 607ae453221d422e9d07882830c02275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:12.395099 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b22d207fbc234913b41a781c8836419e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.395432 1767 tablet_bootstrap.cc:492] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:12.395573 1767 ts_tablet_manager.cc:1403] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20251024 19:48:12.396389 1767 raft_consensus.cc:359] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.396562 1767 raft_consensus.cc:385] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.396613 1767 raft_consensus.cc:740] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:12.396857 1767 consensus_queue.cc:260] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.397327 1767 ts_tablet_manager.cc:1434] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:12.397624 1767 tablet_bootstrap.cc:492] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:12.397967 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c694a372afc94f89972c972793c5e901. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.398049 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 583d0ef19aad4a80a8da15f7abc481c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.389911 1477 tablet_service.cc:1505] Processing CreateTablet for tablet d0cbe0a3436a498a84c892ac156798ed (DEFAULT_TABLE table=default.Table_4dca2888019448ca97c68a8c6dd8396a [id=9da07f2aef964c9a838808d27f211636]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.401533 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da699a314ce4984b7566145ffb74ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.401986 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cbe0a3436a498a84c892ac156798ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.402845 1767 tablet_bootstrap.cc:654] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:12.402622 1185 catalog_manager.cc:5649] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:12.403095 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 187925082cdd4923a6e9148bf9f270fd (table default.Table_55f8c66bd97343438051a98af12d8c2c [id=33f5ac73298543d0b5af77a27e897b79]). Expected version 1 got 0
I20251024 19:48:12.403515 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet a3259823b622461fa823841925b3d203 (table default.Table_ea0d3589c13a4ebeb950ddc737a9948b [id=350b37bf42194fcfbb660bffe83e47c4]). Expected version 4 got 0
I20251024 19:48:12.403733 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 07e75c7f35544bec82cb994dea5ee4ef (table default.Table_83b4409228694c939a4a734c3b6bdd76 [id=450dc07c78d343aea5c8c2d2a6e26444]). Expected version 3 got 0
I20251024 19:48:12.403952 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 7503b22d9e12481b9af96385b9d76e50 (table default.Table_e5c532eb7fc241039c7e06e4757ea011 [id=9065c680f4da42b5a6e5ca335c8e2f3b]). Expected version 3 got 0
I20251024 19:48:12.404628 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37163451258c436ab1f874dd0d66d393. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.408797 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9275741737ba44e69b18a9419a10d6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.412941 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 9275741737ba44e69b18a9419a10d6a6 (DEFAULT_TABLE table=default.Table_092242c51bbb4695947456a8ea5513dc [id=65830805c2cc488fa128ce367145ec2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.415663 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104176d304384d1b99b051ae5744eec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.418627 1774 ts_tablet_manager.cc:1929] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:12.418764 1774 log.cc:1199] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/5b96fc2309274048bed7a35e4bf4bae2
I20251024 19:48:12.419143 1774 ts_tablet_manager.cc:1950] T 5b96fc2309274048bed7a35e4bf4bae2 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:12.421759 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b22d207fbc234913b41a781c8836419e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.425767 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cbe0a3436a498a84c892ac156798ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.429688 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 182314f0f3074150b139cb2476976e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.431234 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9275741737ba44e69b18a9419a10d6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.417032 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 60fbfb1620934034af2acaa39677554f (DEFAULT_TABLE table=default.Table_df275b460e8245c8bc231bab1fb44630 [id=08c466bccc854f82a9da4212e7ce813f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.417119 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 2764689179504e079924fde7e425dbcd (DEFAULT_TABLE table=default.Table_e63d3e693e1148a7b8d2b5731b1027bc [id=5fc5a169166b436ba9343a652a650b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.417249 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 1da699a314ce4984b7566145ffb74ef1 (DEFAULT_TABLE table=default.Table_475f0aa37b7b4cff876223e3d8a0514c [id=d4dbe7cda31e4901b0e1aed96d5670ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.423878 1476 tablet_service.cc:1505] Processing CreateTablet for tablet c8147e8a6e0141669f109b41a8319398 (DEFAULT_TABLE table=default.Table_96fd924db35c4970944db55c46bace57 [id=189e9845121c4f22a7a1f16a581fadaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.424332 1472 tablet_service.cc:1505] Processing CreateTablet for tablet 2c9ff67824254c04bd7d024cbcc6c080 (DEFAULT_TABLE table=default.Table_b1bf5228ae5f485abeb61ea7d512b30d [id=a38b28fa71b14c41b090b6c312ada1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.424782 1480 tablet_service.cc:1505] Processing CreateTablet for tablet fd6aa122a7504daba11aea3e7f507389 (DEFAULT_TABLE table=default.Table_3253b4bae4f14474b0c319b84e6b3130 [id=b0eaaa50b6174d188711550594d42ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.425211 1485 tablet_service.cc:1505] Processing CreateTablet for tablet ec11453f49054f9dbbe620374e59a6ce (DEFAULT_TABLE table=default.Table_abc327da99a44b9d9f01f12c54b1ede2 [id=d0ed1317be5c4d6d915e0ff881de5acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.433513 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 10a19c15fc0d4e3e82373c51c59d18fd (DEFAULT_TABLE table=default.Table_281f4fc881c345e7b86cdc489422126b [id=e0c418ba3dd54e889b0bdbd0949bec1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.434556 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd6aa122a7504daba11aea3e7f507389. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.436600 2640 raft_consensus.cc:493] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.436772 2640 raft_consensus.cc:515] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.437669 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec11453f49054f9dbbe620374e59a6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.438083 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c9ff67824254c04bd7d024cbcc6c080. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.438091 2640 leader_election.cc:290] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:12.440186 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_09f15fac93da45ab80457bd1efc49464"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.440789 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09f15fac93da45ab80457bd1efc49464 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.441248 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10a19c15fc0d4e3e82373c51c59d18fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.441870 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2764689179504e079924fde7e425dbcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.445679 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8147e8a6e0141669f109b41a8319398. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.449851 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9ea29ee3999043faa320834c6c8f12b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.450491 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ea29ee3999043faa320834c6c8f12b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.451156 1768 tablet_bootstrap.cc:492] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:12.451287 1768 ts_tablet_manager.cc:1403] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.139s user 0.007s sys 0.000s
I20251024 19:48:12.451704 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_44475cd37f524126a9ac72ec03a5712c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:12.452085 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e3d3cfc2bb044e84acb9624615065375"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.452283 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44475cd37f524126a9ac72ec03a5712c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:12.452596 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3d3cfc2bb044e84acb9624615065375 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.452654 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_33fc19a822a142afb669502687f7980c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.453097 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33fc19a822a142afb669502687f7980c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.453600 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e97c30902b564ba2a411ea987cb0ef70" } new_table_name: "default.Table_d56f14204c1b485fa6dffd55f3e7c95c" modify_external_catalogs: true
I20251024 19:48:12.454126 1768 raft_consensus.cc:359] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.454572 1768 raft_consensus.cc:385] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.454751 1768 raft_consensus.cc:740] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:12.455283 1768 consensus_queue.cc:260] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.456761 1768 ts_tablet_manager.cc:1434] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251024 19:48:12.457077 2640 raft_consensus.cc:493] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.457485 2640 raft_consensus.cc:515] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.458042 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4243f11bf2b34b30b10766c19e2c6409" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:12.459899 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60fbfb1620934034af2acaa39677554f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.461833 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9bfa6cbfde5148d78028b23509294e53" } new_table_name: "default.Table_f9f3cd8a1e5748f6a9d6b77180996272" modify_external_catalogs: true
I20251024 19:48:12.462474 2640 leader_election.cc:290] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:12.464103 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_147e888401aa40e5a60d97884b36cb98" } modify_external_catalogs: true
W20251024 19:48:12.466679 1586 leader_election.cc:343] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:12.466758 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07e75c7f35544bec82cb994dea5ee4ef" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:12.468427 1349 tablet_service.cc:1505] Processing CreateTablet for tablet c694a372afc94f89972c972793c5e901 (DEFAULT_TABLE table=default.Table_43dbb020b49b44cfa627ef85edba938c [id=7d5fa7422f654970b1eee19e5f6d4360]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.469125 1338 tablet_service.cc:1505] Processing CreateTablet for tablet f709db684b6c49cbaeef3bf8d76f5828 (DEFAULT_TABLE table=default.Table_5bfbf5d7f5764b96a95bf3ac0e0aba69 [id=9a49120c5b654676a63cd3966eaf31a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.469516 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_147e888401aa40e5a60d97884b36cb98" } modify_external_catalogs: true
W20251024 19:48:12.472988 1586 leader_election.cc:343] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:12.477344 2767 log.cc:927] Time spent T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20251024 19:48:12.480123 2441 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-0/data/info.pb
W20251024 19:48:12.481696 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 839cef50795840fab564b94d88565f8c consensus state has no leader
I20251024 19:48:12.482251 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4243f11bf2b34b30b10766c19e2c6409" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:12.483014 2769 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.252.254:36505 every 8 connection(s)
I20251024 19:48:12.483965 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c694a372afc94f89972c972793c5e901. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.484436 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da699a314ce4984b7566145ffb74ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.486182 1011 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskc9NInd/build/asan/bin/kudu as pid 2441
W20251024 19:48:12.490911 1586 leader_election.cc:343] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:12.491044 1586 leader_election.cc:304] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:12.491708 2569 raft_consensus.cc:2749] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:12.492261 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 520060a9adca4bb0b8616bcc006dce52 (DEFAULT_TABLE table=default.Table_9e7fa6339ee24bab9d78fd7c318ed78d [id=b03b545fcd2f47168d4309e5a6c5ea2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.492663 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 520060a9adca4bb0b8616bcc006dce52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.496685 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 1d04a9b54bd6446d8deb64292cef672d
I20251024 19:48:12.381547 1775 ts_tablet_manager.cc:1950] T 3754540ae21e4c65b5af3d0aa3f9189f P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:12.425642 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 63518b583ac94fa6abde420e176ec8ba (DEFAULT_TABLE table=default.Table_ce9762cc2ae44e2dbf2404da0554444c [id=47ae0764c00249b78055e4a5beb29248]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.498034 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63518b583ac94fa6abde420e176ec8ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:12.500046 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a99e9b3fe044c7eb14fa9bffa903873 consensus state has no leader
I20251024 19:48:12.501097 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 10a19c15fc0d4e3e82373c51c59d18fd (DEFAULT_TABLE table=default.Table_281f4fc881c345e7b86cdc489422126b [id=e0c418ba3dd54e889b0bdbd0949bec1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.501489 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10a19c15fc0d4e3e82373c51c59d18fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.426097 1474 tablet_service.cc:1505] Processing CreateTablet for tablet fc35c0a2b36c45ffa2fe4ad00abccfd1 (DEFAULT_TABLE table=default.Table_35b42b8f59b042ba9161297fca0d9bd3 [id=ddd649b907b64761a8ad29ba023d9a32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.501854 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc35c0a2b36c45ffa2fe4ad00abccfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.505867 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 2764689179504e079924fde7e425dbcd (DEFAULT_TABLE table=default.Table_e63d3e693e1148a7b8d2b5731b1027bc [id=5fc5a169166b436ba9343a652a650b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.506201 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2764689179504e079924fde7e425dbcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.510285 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 60fbfb1620934034af2acaa39677554f (DEFAULT_TABLE table=default.Table_df275b460e8245c8bc231bab1fb44630 [id=08c466bccc854f82a9da4212e7ce813f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.510629 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60fbfb1620934034af2acaa39677554f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.426884 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 9f791083b4964ff9b0f5fc575478892c (DEFAULT_TABLE table=default.Table_867f1b66094349fa8d81af1e89bebb06 [id=3959b7b1ee654e149138eb210c3455a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.514495 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f791083b4964ff9b0f5fc575478892c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.427141 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 424bce2f496b4a4fa74e57794f6a11eb (DEFAULT_TABLE table=default.Table_a478a67f294e4fdcba8d2f2b421d26fd [id=2e0492e797864318bb9bfa9ffce2be31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.522683 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 424bce2f496b4a4fa74e57794f6a11eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.523105 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f709db684b6c49cbaeef3bf8d76f5828. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.524348 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 639) took 1287 ms. Trace:
I20251024 19:48:12.524441 1774 rpcz_store.cc:276] 1024 19:48:11.237288 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.237621 (+ 333us) service_pool.cc:225] Handling call
1024 19:48:12.219289 (+981668us) ts_tablet_manager.cc:1151] Deleting tablet 5b96fc2309274048bed7a35e4bf4bae2
1024 19:48:12.219299 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.338700 (+119401us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.418492 (+ 79792us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.524327 (+105835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":981524}
I20251024 19:48:12.524799 1774 tablet_replica.cc:333] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:12.524966 1774 raft_consensus.cc:2243] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:12.525100 1774 pending_rounds.cc:70] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:12.525141 1774 pending_rounds.cc:77] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7214429353699024896 op_type: NO_OP noop_request { }
I20251024 19:48:12.525238 1774 raft_consensus.cc:2889] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:12.525313 1774 raft_consensus.cc:2272] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:12.427378 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 95e11ed28aab4e809e8fc1615f21744c (DEFAULT_TABLE table=default.Table_babcfed7fd5b4b6ebf17c81871843326 [id=288378453f244ce3aae05bbd428e0ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.525976 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95e11ed28aab4e809e8fc1615f21744c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.427642 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 22fa56d227f54eaf9f58ca116e730931 (DEFAULT_TABLE table=default.Table_aa1ad4ed1e214bb890edff3ebe855022 [id=7a58cfd8391e47439763a78cd17af696]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.530777 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fa56d227f54eaf9f58ca116e730931. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.414739 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 520060a9adca4bb0b8616bcc006dce52 (DEFAULT_TABLE table=default.Table_9e7fa6339ee24bab9d78fd7c318ed78d [id=b03b545fcd2f47168d4309e5a6c5ea2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.533730 2656 raft_consensus.cc:493] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.533892 2656 raft_consensus.cc:515] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:12.534034 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 520060a9adca4bb0b8616bcc006dce52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.534631 2656 leader_election.cc:290] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:12.534780 2656 raft_consensus.cc:493] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.536420 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c7dbe9ff2c4869994c4f43d5676c30" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:12.537119 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 9f791083b4964ff9b0f5fc575478892c (DEFAULT_TABLE table=default.Table_867f1b66094349fa8d81af1e89bebb06 [id=3959b7b1ee654e149138eb210c3455a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.537725 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f791083b4964ff9b0f5fc575478892c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:12.539634 1444 consensus_peers.cc:597] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 913534ab88dd459e9cac2faebb5492a1. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:12.540434 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 488) took 6396 ms. Trace:
I20251024 19:48:12.540520 1775 rpcz_store.cc:276] 1024 19:48:06.144104 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.145205 (+ 1101us) service_pool.cc:225] Handling call
1024 19:48:12.110765 (+5965560us) ts_tablet_manager.cc:1151] Deleting tablet 3754540ae21e4c65b5af3d0aa3f9189f
1024 19:48:12.110778 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.165322 (+ 54544us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.338083 (+172761us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.540405 (+202322us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5903529}
I20251024 19:48:12.540993 1775 tablet_replica.cc:333] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:12.541218 1775 raft_consensus.cc:2243] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:12.541452 1775 raft_consensus.cc:2272] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:12.542057 1775 ts_tablet_manager.cc:1916] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:12.542291 2656 raft_consensus.cc:515] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.543401 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdfa5eea3b674a3389a156d1c72204df" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:12.551245 1767 tablet_bootstrap.cc:492] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:12.555725 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 3754540ae21e4c65b5af3d0aa3f9189f (table default.Table_eb1e63c7ef2244259c8a64fd4fe3859b [id=fec3ddedec0b44db80b30c899043bc15]) successfully deleted
I20251024 19:48:12.555836 1474 tablet_service.cc:1505] Processing CreateTablet for tablet b383a83425654723bd1d9c3a39b187be (DEFAULT_TABLE table=default.Table_09f15fac93da45ab80457bd1efc49464 [id=47ed56aa60974fef835ce3cc8d3a713d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.556329 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b383a83425654723bd1d9c3a39b187be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.556327 1767 ts_tablet_manager.cc:1403] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.159s user 0.015s sys 0.030s
I20251024 19:48:12.556836 1774 ts_tablet_manager.cc:1916] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:12.557304 1767 raft_consensus.cc:359] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.557514 1767 raft_consensus.cc:385] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.559078 1767 raft_consensus.cc:740] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:12.551992 1776 ts_tablet_manager.cc:1929] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:12.559486 1776 log.cc:1199] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/fdfa5eea3b674a3389a156d1c72204df
I20251024 19:48:12.559876 1776 ts_tablet_manager.cc:1950] T fdfa5eea3b674a3389a156d1c72204df P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:12.561385 2640 consensus_queue.cc:1048] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:12.561748 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d04a9b54bd6446d8deb64292cef672d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:12.561546 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 704) took 1215 ms. Trace:
I20251024 19:48:12.562088 1776 rpcz_store.cc:276] 1024 19:48:11.346537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.346826 (+ 289us) service_pool.cc:225] Handling call
1024 19:48:12.288203 (+941377us) ts_tablet_manager.cc:1151] Deleting tablet fdfa5eea3b674a3389a156d1c72204df
1024 19:48:12.288212 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.472938 (+184726us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.551898 (+ 78960us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.561526 (+ 9628us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":941184}
I20251024 19:48:12.562744 1776 tablet_replica.cc:333] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:12.562934 1776 raft_consensus.cc:2243] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:12.563196 1776 raft_consensus.cc:2272] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:12.563709 1776 ts_tablet_manager.cc:1916] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:12.559854 1767 consensus_queue.cc:260] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.571875 1767 ts_tablet_manager.cc:1434] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251024 19:48:12.572844 1767 tablet_bootstrap.cc:492] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:12.574923 1767 tablet_bootstrap.cc:654] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:12.577148 2656 leader_election.cc:290] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:12.579860 1768 tablet_bootstrap.cc:492] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:12.586483 1766 tablet_bootstrap.cc:492] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:12.586700 1766 ts_tablet_manager.cc:1403] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.255s user 0.002s sys 0.003s
I20251024 19:48:12.587630 1766 raft_consensus.cc:359] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.587744 1368 raft_consensus.cc:3060] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:12.587940 1766 raft_consensus.cc:385] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.588066 1766 raft_consensus.cc:740] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:12.588434 1766 consensus_queue.cc:260] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.589109 1766 ts_tablet_manager.cc:1434] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:12.589790 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 63518b583ac94fa6abde420e176ec8ba (DEFAULT_TABLE table=default.Table_ce9762cc2ae44e2dbf2404da0554444c [id=47ae0764c00249b78055e4a5beb29248]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.589980 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 2c9ff67824254c04bd7d024cbcc6c080 (DEFAULT_TABLE table=default.Table_b1bf5228ae5f485abeb61ea7d512b30d [id=a38b28fa71b14c41b090b6c312ada1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.590178 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63518b583ac94fa6abde420e176ec8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.590116 1337 tablet_service.cc:1505] Processing CreateTablet for tablet b383a83425654723bd1d9c3a39b187be (DEFAULT_TABLE table=default.Table_09f15fac93da45ab80457bd1efc49464 [id=47ed56aa60974fef835ce3cc8d3a713d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.590476 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b383a83425654723bd1d9c3a39b187be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.590420 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 22fa56d227f54eaf9f58ca116e730931 (DEFAULT_TABLE table=default.Table_aa1ad4ed1e214bb890edff3ebe855022 [id=7a58cfd8391e47439763a78cd17af696]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.590683 1341 tablet_service.cc:1505] Processing CreateTablet for tablet fc35c0a2b36c45ffa2fe4ad00abccfd1 (DEFAULT_TABLE table=default.Table_35b42b8f59b042ba9161297fca0d9bd3 [id=ddd649b907b64761a8ad29ba023d9a32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.590767 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c9ff67824254c04bd7d024cbcc6c080. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.590957 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc35c0a2b36c45ffa2fe4ad00abccfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.591041 1349 tablet_service.cc:1505] Processing CreateTablet for tablet fd6aa122a7504daba11aea3e7f507389 (DEFAULT_TABLE table=default.Table_3253b4bae4f14474b0c319b84e6b3130 [id=b0eaaa50b6174d188711550594d42ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.591019 2640 consensus_queue.cc:1048] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:12.591306 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd6aa122a7504daba11aea3e7f507389. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.591439 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d04a9b54bd6446d8deb64292cef672d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:12.592080 1766 tablet_bootstrap.cc:492] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:12.594048 1766 tablet_bootstrap.cc:654] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:12.594202 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 424bce2f496b4a4fa74e57794f6a11eb (DEFAULT_TABLE table=default.Table_a478a67f294e4fdcba8d2f2b421d26fd [id=2e0492e797864318bb9bfa9ffce2be31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.594596 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 424bce2f496b4a4fa74e57794f6a11eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.595475 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fa56d227f54eaf9f58ca116e730931. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.595484 1340 tablet_service.cc:1505] Processing CreateTablet for tablet ec11453f49054f9dbbe620374e59a6ce (DEFAULT_TABLE table=default.Table_abc327da99a44b9d9f01f12c54b1ede2 [id=d0ed1317be5c4d6d915e0ff881de5acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.595834 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec11453f49054f9dbbe620374e59a6ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:12.600214 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 44470.9us
I20251024 19:48:12.549186 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07e75c7f35544bec82cb994dea5ee4ef" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:12.612634 1369 raft_consensus.cc:3060] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:12.618654 1775 ts_tablet_manager.cc:1929] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:12.618798 1775 log.cc:1199] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/0a12e415590a4c71aab4ad7d30412c83
I20251024 19:48:12.619201 1775 ts_tablet_manager.cc:1950] T 0a12e415590a4c71aab4ad7d30412c83 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:12.620208 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 489) took 6347 ms. Trace:
I20251024 19:48:12.620312 1775 rpcz_store.cc:276] 1024 19:48:06.272409 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.272488 (+ 79us) service_pool.cc:225] Handling call
1024 19:48:12.540939 (+6268451us) ts_tablet_manager.cc:1151] Deleting tablet 0a12e415590a4c71aab4ad7d30412c83
1024 19:48:12.540948 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.545186 (+ 4238us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.618514 (+ 73328us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.620186 (+ 1672us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6268138}
I20251024 19:48:12.620867 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 494) took 6253 ms. Trace:
I20251024 19:48:12.620959 1775 rpcz_store.cc:276] 1024 19:48:06.367489 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.367564 (+ 75us) service_pool.cc:225] Handling call
1024 19:48:12.620737 (+6253173us) ts_tablet_manager.cc:1151] Deleting tablet 3324755a5a184a609fae5f7a25d58e4c
1024 19:48:12.620748 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.620850 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6237458}
I20251024 19:48:12.621348 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 492) took 6254 ms. Trace:
W20251024 19:48:12.621515 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 8d1b6eaa353c4284a2b114b42437272c (table default.Table_673109e8b8e744f99120d9f081b1dd9d [id=51f5913271754b6f86525a41d9a0ce22], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8d1b6eaa353c4284a2b114b42437272c consensus state has no leader
W20251024 19:48:12.622568 1447 consensus_peers.cc:597] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:12.622648 1766 tablet_bootstrap.cc:492] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:12.622761 1766 ts_tablet_manager.cc:1403] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20251024 19:48:12.623576 2615 consensus_queue.cc:1048] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:12.624080 1766 raft_consensus.cc:359] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.624243 1766 raft_consensus.cc:385] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.624296 1766 raft_consensus.cc:740] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:12.624517 1766 consensus_queue.cc:260] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.625276 1776 ts_tablet_manager.cc:1929] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:12.625478 1776 log.cc:1199] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/59c7dbe9ff2c4869994c4f43d5676c30
I20251024 19:48:12.625972 1776 ts_tablet_manager.cc:1950] T 59c7dbe9ff2c4869994c4f43d5676c30 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:12.627808 1766 ts_tablet_manager.cc:1434] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251024 19:48:12.628118 1766 tablet_bootstrap.cc:492] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:12.629891 2776 sys_catalog.cc:263] Verifying existing consensus state
I20251024 19:48:12.621438 1775 rpcz_store.cc:276] 1024 19:48:06.367261 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.367712 (+ 451us) service_pool.cc:225] Handling call
1024 19:48:12.621202 (+6253490us) ts_tablet_manager.cc:1151] Deleting tablet 780442b310e6409498e6bbe4256280a4
1024 19:48:12.621214 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.621330 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6237010}
I20251024 19:48:12.633699 2615 raft_consensus.cc:493] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.633879 2615 raft_consensus.cc:515] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.634341 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 493) took 6266 ms. Trace:
I20251024 19:48:12.634609 1775 rpcz_store.cc:276] 1024 19:48:06.367387 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.367933 (+ 546us) service_pool.cc:225] Handling call
1024 19:48:12.633829 (+6265896us) ts_tablet_manager.cc:1151] Deleting tablet 65424b00ab9c454f9584048caba83e6a
1024 19:48:12.633843 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.634233 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6249504}
I20251024 19:48:12.635269 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "607ae453221d422e9d07882830c02275" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:12.635480 1650 raft_consensus.cc:2468] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:12.635915 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 495) took 6268 ms. Trace:
I20251024 19:48:12.635972 1444 leader_election.cc:304] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:12.636449 2743 raft_consensus.cc:2804] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:12.636533 2743 raft_consensus.cc:493] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:12.636622 2743 raft_consensus.cc:3060] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:12.636735 2776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Bootstrap starting.
I20251024 19:48:12.637918 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 702) took 1291 ms. Trace:
I20251024 19:48:12.638013 1776 rpcz_store.cc:276] 1024 19:48:11.346315 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.346958 (+ 643us) service_pool.cc:225] Handling call
1024 19:48:12.562693 (+1215735us) ts_tablet_manager.cc:1151] Deleting tablet 59c7dbe9ff2c4869994c4f43d5676c30
1024 19:48:12.562705 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.611528 (+ 48823us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.625115 (+ 13587us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.637880 (+ 12765us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1215509}
I20251024 19:48:12.638617 1776 tablet_replica.cc:333] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:12.638828 1776 raft_consensus.cc:2243] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:12.639082 1776 pending_rounds.cc:70] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:12.639122 1776 pending_rounds.cc:77] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429354959523840 op_type: NO_OP noop_request { }
I20251024 19:48:12.639218 1776 raft_consensus.cc:2889] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:12.639294 1776 raft_consensus.cc:2272] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:12.639322 2743 raft_consensus.cc:515] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.639918 2743 leader_election.cc:290] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:12.640204 1776 ts_tablet_manager.cc:1916] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:12.640906 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "607ae453221d422e9d07882830c02275" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:12.641064 1650 raft_consensus.cc:3060] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:12.644336 1650 raft_consensus.cc:2468] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:12.645860 1444 leader_election.cc:304] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:12.634634 2615 leader_election.cc:290] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:12.647047 2615 raft_consensus.cc:2804] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:12.647177 2615 raft_consensus.cc:697] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:12.647449 2615 consensus_queue.cc:237] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:12.650465 1447 consensus_peers.cc:597] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:12.651394 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "607ae453221d422e9d07882830c02275" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:12.651819 1776 ts_tablet_manager.cc:1929] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:12.651875 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 4ffd446c269448d38cc8d987a53aecf9 (table default.Table_bedd8edc24014953ac8c7dcecf18f128 [id=2ae1c808837e48df97c4efb01a684da9]). Expected version 1 got 0
I20251024 19:48:12.651964 1776 log.cc:1199] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/758df87acbdb4adaa83e1ecbc44e7cf1
I20251024 19:48:12.652173 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet e03c488d0f264684b0897a81a04d7466 (table default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8]). Expected version 2 got 0
I20251024 19:48:12.652184 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "607ae453221d422e9d07882830c02275" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:12.652354 1776 ts_tablet_manager.cc:1950] T 758df87acbdb4adaa83e1ecbc44e7cf1 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
W20251024 19:48:12.652631 1447 leader_election.cc:343] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:12.653142 1447 leader_election.cc:343] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:12.653256 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 707) took 1306 ms. Trace:
I20251024 19:48:12.653342 1776 rpcz_store.cc:276] 1024 19:48:11.346898 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.347052 (+ 154us) service_pool.cc:225] Handling call
1024 19:48:12.638561 (+1291509us) ts_tablet_manager.cc:1151] Deleting tablet 758df87acbdb4adaa83e1ecbc44e7cf1
1024 19:48:12.638572 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.644228 (+ 5656us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.651690 (+ 7462us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.653230 (+ 1540us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42,"tablet-delete.queue_time_us":1291370}
I20251024 19:48:12.653775 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 705) took 1307 ms. Trace:
I20251024 19:48:12.653856 1776 rpcz_store.cc:276] 1024 19:48:11.346623 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.347510 (+ 887us) service_pool.cc:225] Handling call
1024 19:48:12.653687 (+1306177us) ts_tablet_manager.cc:1151] Deleting tablet 2037fcca87b94b5fb31639b4d07307b0
1024 19:48:12.653696 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.653761 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1306006}
I20251024 19:48:12.654091 1776 tablet_replica.cc:333] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:12.654258 1776 raft_consensus.cc:2243] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:12.654518 1776 raft_consensus.cc:2272] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:12.657883 1339 tablet_service.cc:1505] Processing CreateTablet for tablet c8147e8a6e0141669f109b41a8319398 (DEFAULT_TABLE table=default.Table_96fd924db35c4970944db55c46bace57 [id=189e9845121c4f22a7a1f16a581fadaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.658288 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8147e8a6e0141669f109b41a8319398. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.663589 1766 tablet_bootstrap.cc:654] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:12.664981 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 182314f0f3074150b139cb2476976e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.668696 1776 ts_tablet_manager.cc:1916] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:12.670430 2569 raft_consensus.cc:493] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.670711 2569 raft_consensus.cc:515] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.673142 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "187bc968c2b049028dd25dfb0264c0aa" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:12.673806 1586 leader_election.cc:343] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:12.674368 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "187bc968c2b049028dd25dfb0264c0aa" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:12.675307 1586 leader_election.cc:343] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:12.675477 1586 leader_election.cc:304] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:12.675771 2747 raft_consensus.cc:2749] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:12.679795 2569 leader_election.cc:290] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:12.681406 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 95e11ed28aab4e809e8fc1615f21744c (DEFAULT_TABLE table=default.Table_babcfed7fd5b4b6ebf17c81871843326 [id=288378453f244ce3aae05bbd428e0ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.681922 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95e11ed28aab4e809e8fc1615f21744c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:12.684062 1328 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:54172 (request call id 131) took 1099 ms. Trace:
I20251024 19:48:12.684156 1328 rpcz_store.cc:276] 1024 19:48:11.584904 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.584962 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:12.094812 (+509850us) spinlock_profiling.cc:243] Waited 504 ms on lock 0x6160001a2354. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:12.684033 (+589221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1158619392}
I20251024 19:48:12.687110 1768 tablet_bootstrap.cc:654] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:12.694698 1586 leader_election.cc:343] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:12.694955 1586 leader_election.cc:304] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:12.708307 2615 raft_consensus.cc:493] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:12.708470 2615 raft_consensus.cc:515] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.709136 2615 leader_election.cc:290] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:12.711622 1650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4243f11bf2b34b30b10766c19e2c6409" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:12.711894 1650 raft_consensus.cc:2468] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:12.712308 1444 leader_election.cc:304] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:12.712670 2615 raft_consensus.cc:2804] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:12.712764 2615 raft_consensus.cc:493] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:12.712831 2615 raft_consensus.cc:3060] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:12.715816 2615 raft_consensus.cc:515] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.717028 2615 leader_election.cc:290] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:12.723340 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4243f11bf2b34b30b10766c19e2c6409" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:12.723558 1648 raft_consensus.cc:3060] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:12.727630 1648 raft_consensus.cc:2468] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:12.728391 1444 leader_election.cc:304] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:12.728914 2567 raft_consensus.cc:493] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af3952de050140d383f6b0de7666fa97)
I20251024 19:48:12.729107 2567 raft_consensus.cc:515] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.730007 2567 leader_election.cc:290] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:12.730175 2567 raft_consensus.cc:2804] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:12.730297 2567 raft_consensus.cc:697] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:12.730629 2567 consensus_queue.cc:237] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:12.734391 1444 leader_election.cc:343] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 5b96fc2309274048bed7a35e4bf4bae2
I20251024 19:48:12.741406 1776 ts_tablet_manager.cc:1929] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:12.743112 1776 log.cc:1199] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/dc14b0cdb7f342ac956a7857ab3f35d7
I20251024 19:48:12.748390 1776 ts_tablet_manager.cc:1950] T dc14b0cdb7f342ac956a7857ab3f35d7 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:12.754819 2569 raft_consensus.cc:2749] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:12.636346 1775 rpcz_store.cc:276] 1024 19:48:06.367635 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.368094 (+ 459us) service_pool.cc:225] Handling call
1024 19:48:12.635507 (+6267413us) ts_tablet_manager.cc:1151] Deleting tablet 7eb62791100f477784ab315923b6ceb6
1024 19:48:12.635518 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.635896 (+ 378us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6251132}
I20251024 19:48:12.770082 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_acc74be61adc4422bd3935ae750cf797"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.770939 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acc74be61adc4422bd3935ae750cf797 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.782548 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_74e6a6d8b7a34049994d41a379a9a9f0" } modify_external_catalogs: true
I20251024 19:48:12.782949 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_74e6a6d8b7a34049994d41a379a9a9f0" } modify_external_catalogs: true
I20251024 19:48:12.786489 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 498) took 6330 ms. Trace:
I20251024 19:48:12.786931 1775 rpcz_store.cc:276] 1024 19:48:06.436191 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.437886 (+ 1695us) service_pool.cc:225] Handling call
1024 19:48:12.766544 (+6328658us) ts_tablet_manager.cc:1151] Deleting tablet 94b35252e63f4a8ba305cc029ddb5ae8
1024 19:48:12.766607 (+ 63us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.766970 (+ 363us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6322223}
I20251024 19:48:12.787923 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 499) took 6347 ms. Trace:
I20251024 19:48:12.788775 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_43dbb020b49b44cfa627ef85edba938c" } new_table_name: "default.Table_c125d060e7bb4283b4ee1bf43680f6b9" modify_external_catalogs: true
I20251024 19:48:12.795946 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:54172 (request call id 130) took 1274 ms. Trace:
I20251024 19:48:12.796073 1332 rpcz_store.cc:276] 1024 19:48:11.521563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.521643 (+ 80us) service_pool.cc:225] Handling call
1024 19:48:12.091155 (+569512us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x6160001a2354. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:12.795912 (+704757us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1182056704}
I20251024 19:48:12.802182 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 64325d89db714554866e74922d103e34
I20251024 19:48:12.805850 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0e854ff9b0b84dcbbdea3ec2c97a7aef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.806473 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e854ff9b0b84dcbbdea3ec2c97a7aef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.812598 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_894d6739340148a995dd849a205c19c3" } new_table_name: "default.Table_1a079b981b694617b3baf9897dfb37ae" modify_external_catalogs: true
W20251024 19:48:12.803035 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c694a372afc94f89972c972793c5e901 consensus state has no leader
W20251024 19:48:12.823522 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6de2c21796f4bc4904f88b654c3baec consensus state has no leader
I20251024 19:48:12.828217 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4da9eaf31f3d4d80870061b13b95cad5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.829013 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4da9eaf31f3d4d80870061b13b95cad5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.831076 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2df2e83bbab54dcabedf78455d66e990"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:12.831583 1768 tablet_bootstrap.cc:492] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
W20251024 19:48:12.831667 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2df2e83bbab54dcabedf78455d66e990 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.831732 1768 ts_tablet_manager.cc:1403] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.252s user 0.009s sys 0.005s
I20251024 19:48:12.833457 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b2510c82f11444d78f61be9b9f2fcaff" } modify_external_catalogs: true
I20251024 19:48:12.833678 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b2510c82f11444d78f61be9b9f2fcaff" } modify_external_catalogs: true
I20251024 19:48:12.838457 1768 raft_consensus.cc:359] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:12.838721 1768 raft_consensus.cc:385] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.838775 1768 raft_consensus.cc:740] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:12.839138 1768 consensus_queue.cc:260] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:12.839679 1768 ts_tablet_manager.cc:1434] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251024 19:48:12.839989 1768 tablet_bootstrap.cc:492] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:12.843767 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_628b2afa9401426980be0eae5dffd917" } modify_external_catalogs: true
I20251024 19:48:12.844125 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_628b2afa9401426980be0eae5dffd917" } modify_external_catalogs: true
I20251024 19:48:12.847828 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f5ee199153824340833717cac0a3cea0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.848826 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5ee199153824340833717cac0a3cea0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.802569 2615 consensus_queue.cc:1048] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:12.804245 1557 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:12.852823 1768 tablet_bootstrap.cc:654] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:12.853844 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4243f11bf2b34b30b10766c19e2c6409" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:12.853948 1775 rpcz_store.cc:276] 1024 19:48:06.440407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.440456 (+ 49us) service_pool.cc:225] Handling call
1024 19:48:12.787772 (+6347316us) ts_tablet_manager.cc:1151] Deleting tablet 83b91a5d764449d5b82ef6d1ee0360f5
1024 19:48:12.787786 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.787905 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6347115}
I20251024 19:48:12.854523 1775 tablet_replica.cc:333] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:12.854708 1775 raft_consensus.cc:2243] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:12.855057 1775 raft_consensus.cc:2272] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:12.856024 1775 ts_tablet_manager.cc:1916] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:12.857942 1586 consensus_peers.cc:597] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:12.860416 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a17ef21a2da140b4954cf16abc6d2e95" } new_table_name: "default.Table_003cb0c156394f9b8b0ba1b4ac52ce4c" modify_external_catalogs: true
I20251024 19:48:12.872627 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4243f11bf2b34b30b10766c19e2c6409" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:12.875733 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2a9850f5d5dc499f87e87bdb0efcc93a" } modify_external_catalogs: true
I20251024 19:48:12.876670 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2a9850f5d5dc499f87e87bdb0efcc93a" } modify_external_catalogs: true
W20251024 19:48:12.881906 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6bebcdb80476423b8e6c750f749a1771 consensus state has no leader
I20251024 19:48:12.889449 2640 consensus_queue.cc:1048] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251024 19:48:12.891624 1557 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:12.893980 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8638421b2d894a4dbf4a9f06afc4c595"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.894958 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8638421b2d894a4dbf4a9f06afc4c595 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.896006 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 706) took 1549 ms. Trace:
I20251024 19:48:12.896252 1776 rpcz_store.cc:276] 1024 19:48:11.346722 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.347735 (+ 1013us) service_pool.cc:225] Handling call
1024 19:48:12.654052 (+1306317us) ts_tablet_manager.cc:1151] Deleting tablet dc14b0cdb7f342ac956a7857ab3f35d7
1024 19:48:12.654060 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.682357 (+ 28297us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.740234 (+ 57877us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.895947 (+155713us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":141673,"tablet-delete.queue_time_us":1306118}
I20251024 19:48:12.900919 2796 consensus_queue.cc:1048] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:12.923966 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d61970c7cdd9475d9d6da8dc5f80a232"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.924739 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d61970c7cdd9475d9d6da8dc5f80a232 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.927011 1767 tablet_bootstrap.cc:492] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:12.927248 1767 ts_tablet_manager.cc:1403] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.355s user 0.028s sys 0.037s
I20251024 19:48:12.928459 1767 raft_consensus.cc:359] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.928951 1767 raft_consensus.cc:385] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:12.929255 1767 raft_consensus.cc:740] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:12.929639 1767 consensus_queue.cc:260] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.930405 1767 ts_tablet_manager.cc:1434] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:12.932547 2561 consensus_queue.cc:1048] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:12.933007 1767 tablet_bootstrap.cc:492] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:12.935140 1767 tablet_bootstrap.cc:654] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:12.944664 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet d731403fbe0b4f92ac5899e13ea7bb19
I20251024 19:48:12.948726 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 769f576810ab49368eb0fc7837b7da58
I20251024 19:48:12.954031 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet fce7de9523fb4878a58aa8aef0c6b182
I20251024 19:48:12.967299 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5b54ca8cd83f49188876e061f70d629e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.968377 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b54ca8cd83f49188876e061f70d629e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.971434 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6b101c11690c47319e38598ef667655b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.972240 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b101c11690c47319e38598ef667655b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.974776 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5afc602c5d154cf48565fa17d88d7c68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:12.975518 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5afc602c5d154cf48565fa17d88d7c68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:12.979456 1774 ts_tablet_manager.cc:1929] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
W20251024 19:48:13.021253 1447 leader_election.cc:343] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:13.022011 1447 leader_election.cc:343] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:13.022543 1447 leader_election.cc:343] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 5b96fc2309274048bed7a35e4bf4bae2
I20251024 19:48:13.022639 1447 leader_election.cc:304] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:13.022907 2561 raft_consensus.cc:2749] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251024 19:48:13.023574 1447 consensus_peers.cc:597] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:13.023936 1447 consensus_peers.cc:597] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:13.034735 2615 raft_consensus.cc:493] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:13.034901 2615 raft_consensus.cc:515] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.035463 2615 leader_election.cc:290] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:13.036389 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03cdbd1b62b741249ad78562a7edfd22" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:13.036567 1647 raft_consensus.cc:2468] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:13.037053 1444 leader_election.cc:304] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:13.037554 2796 raft_consensus.cc:2804] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:13.037644 2796 raft_consensus.cc:493] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:13.037703 2796 raft_consensus.cc:3060] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:13.040963 2796 raft_consensus.cc:515] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.042464 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03cdbd1b62b741249ad78562a7edfd22" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:13.042670 1647 raft_consensus.cc:3060] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:13.042683 2796 leader_election.cc:290] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:13.047173 1647 raft_consensus.cc:2468] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:13.047705 1444 leader_election.cc:304] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:13.048100 2796 raft_consensus.cc:2804] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:13.048496 2796 raft_consensus.cc:697] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:13.049041 2796 consensus_queue.cc:237] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.058030 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3bc26ab1c8fd48c682794066b1d61d4e" } new_table_name: "default.Table_d02761e4faeb45c9850d8728eef6a5fa" modify_external_catalogs: true
I20251024 19:48:13.067276 1647 raft_consensus.cc:1275] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 3 FOLLOWER]: Refusing update from remote peer 10a9536207e343f188e0ef709081d04e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251024 19:48:13.067912 2567 consensus_queue.cc:1048] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20251024 19:48:13.068039 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86ff78a220714c4f9ac79f82b44597f6 consensus state has no leader
I20251024 19:48:13.068688 2640 raft_consensus.cc:493] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:13.068822 2640 raft_consensus.cc:515] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.069484 2640 leader_election.cc:290] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:13.075013 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ffd446c269448d38cc8d987a53aecf9" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:13.076177 2776 log.cc:826] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Log is configured to *not* fsync() on all Append() calls
W20251024 19:48:13.077800 1586 leader_election.cc:343] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:13.078228 2640 consensus_queue.cc:1048] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:13.079867 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4aaaf3ef7e68447f8e9c6576c7405f5a" } new_table_name: "default.Table_059135c028074f7fb4bc26da343a4de1" modify_external_catalogs: true
I20251024 19:48:13.083153 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d101352faa2b4f4b9b983143a5142043" } modify_external_catalogs: true
I20251024 19:48:13.083483 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d101352faa2b4f4b9b983143a5142043" } modify_external_catalogs: true
I20251024 19:48:12.897372 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 708) took 1541 ms. Trace:
I20251024 19:48:13.084888 1776 rpcz_store.cc:276] 1024 19:48:11.355456 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.355514 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:12.897041 (+1541527us) ts_tablet_manager.cc:1151] Deleting tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9
1024 19:48:12.897055 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.897314 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1540938}
I20251024 19:48:13.085341 1776 tablet_replica.cc:333] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:13.087576 1189 master_service.cc:954] ReplaceTablet: received request to replace tablet ad055dc3387b414bb55d09b9683ba17e from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:13.091181 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet b22b58404b8b441ba97ef536885a66e0 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:13.100355 1766 tablet_bootstrap.cc:492] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:13.100836 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ad055dc3387b414bb55d09b9683ba17e
I20251024 19:48:13.101174 1766 ts_tablet_manager.cc:1403] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.473s user 0.035s sys 0.070s
I20251024 19:48:13.101964 1189 catalog_manager.cc:6719] ReplaceTablet: tablet ad055dc3387b414bb55d09b9683ba17e deleted and replaced by tablet e83527f7ff5c4e248f344a84515a5d6a
I20251024 19:48:13.102082 2615 raft_consensus.cc:493] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:13.101330 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b22b58404b8b441ba97ef536885a66e0
I20251024 19:48:13.102209 2615 raft_consensus.cc:515] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.102768 1186 catalog_manager.cc:6719] ReplaceTablet: tablet b22b58404b8b441ba97ef536885a66e0 deleted and replaced by tablet f37a01dc3d5f458d81605f2c41fa27dd
I20251024 19:48:13.102859 2615 leader_election.cc:290] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:13.103178 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74029a7d7cdc4503a833c1758887b684 consensus state has no leader
I20251024 19:48:13.103840 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cef80e4d14bb4e719bceebf43041d079" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:13.104079 1649 raft_consensus.cc:2468] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:13.104570 1444 leader_election.cc:304] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:13.104885 2743 raft_consensus.cc:2804] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:13.104964 2743 raft_consensus.cc:493] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:13.105042 2743 raft_consensus.cc:3060] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:13.107623 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet b79801594e95434cbb8fe7b75b14602e (table default.Table_aa7c283553a841e9ab2fa21fb6485a11 [id=7ec9391c52b1424eb52850f2c6ba4779]) successfully deleted
I20251024 19:48:13.107856 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet fdfa5eea3b674a3389a156d1c72204df (table default.Table_73923b0648674b2891cbbf4b90d3f106 [id=96ea5a86f92b4d18b0d74f1092bd17b3]) successfully deleted
I20251024 19:48:13.108047 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 59c7dbe9ff2c4869994c4f43d5676c30 (table default.Table_eb9c8019451d46fba31a87a0b6ac6ad8 [id=e5b1dd31c2114e8a95ba33981029255c]) successfully deleted
I20251024 19:48:13.108229 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 758df87acbdb4adaa83e1ecbc44e7cf1 (table default.Table_003cb0c156394f9b8b0ba1b4ac52ce4c [id=b81a7d71911844c1be495ade2ceaf1d4]) successfully deleted
W20251024 19:48:13.108448 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2037fcca87b94b5fb31639b4d07307b0 because the tablet was not found. No further retry: Not found: Tablet not found: 2037fcca87b94b5fb31639b4d07307b0
I20251024 19:48:13.109267 1776 raft_consensus.cc:2243] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:13.110119 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet 4193750331874d7ba9bbe6279c3a8e37 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:13.110343 1189 master_service.cc:954] ReplaceTablet: received request to replace tablet 7172a93a5bec4fa1b63282ba63beec0c from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:13.110821 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet 639836c656f44a819036f4d00e055bd9 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:13.118764 1490 tablet_service.cc:1505] Processing CreateTablet for tablet dcd06be0791c4a65a1a13edd00dca13b (DEFAULT_TABLE table=default.Table_44475cd37f524126a9ac72ec03a5712c [id=538e81755a22418ab7c8833124cd680a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.121676 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 8eec3aa8f33746c09b63c8f41168eab1 (DEFAULT_TABLE table=default.Table_e3d3cfc2bb044e84acb9624615065375 [id=bd92e84f88c543918f6ad9c91b31d1f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.121809 1482 tablet_service.cc:1505] Processing CreateTablet for tablet f6df45dfab124358b8669bb4f01a6dd5 (DEFAULT_TABLE table=default.Table_33fc19a822a142afb669502687f7980c [id=932ebe916fab4be0b5f2697f11b56105]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:13.122099 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 43951.9us
I20251024 19:48:13.132133 1776 pending_rounds.cc:70] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:13.132205 1776 pending_rounds.cc:77] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429354252419072 op_type: NO_OP noop_request { }
I20251024 19:48:13.132298 1776 raft_consensus.cc:2889] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:13.132388 1776 raft_consensus.cc:2272] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:13.133092 1776 ts_tablet_manager.cc:1916] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:13.136785 2743 raft_consensus.cc:515] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:12.989636 1418 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:13.138088 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 1f5dfcab4dd242c580e2f3214a72837e (DEFAULT_TABLE table=default.Table_9ea29ee3999043faa320834c6c8f12b7 [id=a7ebc3b1f6004cde867cd808f13037be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:12.997830 1372 raft_consensus.cc:1275] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 3 FOLLOWER]: Refusing update from remote peer 10a9536207e343f188e0ef709081d04e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251024 19:48:13.002205 1768 tablet_bootstrap.cc:492] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:13.139544 1768 ts_tablet_manager.cc:1403] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.300s user 0.011s sys 0.020s
I20251024 19:48:13.140106 1774 log.cc:1199] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/b6661b9e9a174849b6093d09cd8c1251
I20251024 19:48:13.072604 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03cdbd1b62b741249ad78562a7edfd22" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:13.143352 1774 ts_tablet_manager.cc:1950] T b6661b9e9a174849b6093d09cd8c1251 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:13.147697 2743 leader_election.cc:290] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:13.148905 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcd06be0791c4a65a1a13edd00dca13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.149163 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f5dfcab4dd242c580e2f3214a72837e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.149369 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eec3aa8f33746c09b63c8f41168eab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.151655 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6df45dfab124358b8669bb4f01a6dd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.152954 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet 57fed826b4b64edba48c5fffaef1e031 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.153293 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet 7172a93a5bec4fa1b63282ba63beec0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.153613 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 64325d89db714554866e74922d103e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:13.153698 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet be8126459ef541f39bc133f27c7d0f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.153853 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet d731403fbe0b4f92ac5899e13ea7bb19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:13.153952 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet da7d38b63708427982591da1014aca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.154181 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet fb08e56db59b4db796c50db5bda93af9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.154512 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 5b4f345b3b974681902f37b3975ea618 (DEFAULT_TABLE table=default.Table_acc74be61adc4422bd3935ae750cf797 [id=51219be242af44fa8fbbe079eb9eb7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.154739 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 182314f0f3074150b139cb2476976e1a (DEFAULT_TABLE table=default.Table_587f4b6c35e94fbca64f79a03aecf9c1 [id=aa7fd0bcc8f141ceb3488072f873b025]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.154983 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b4f345b3b974681902f37b3975ea618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.155078 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 182314f0f3074150b139cb2476976e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.157894 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 769f576810ab49368eb0fc7837b7da58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:13.164090 1767 tablet_bootstrap.cc:492] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:13.164354 1767 ts_tablet_manager.cc:1403] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.231s user 0.016s sys 0.019s
I20251024 19:48:13.106251 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cef80e4d14bb4e719bceebf43041d079" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:13.166745 1767 raft_consensus.cc:359] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:13.166941 1767 raft_consensus.cc:385] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:13.113873 1335 tablet_service.cc:1505] Processing CreateTablet for tablet dcd06be0791c4a65a1a13edd00dca13b (DEFAULT_TABLE table=default.Table_44475cd37f524126a9ac72ec03a5712c [id=538e81755a22418ab7c8833124cd680a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.167032 1767 raft_consensus.cc:740] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:13.167479 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcd06be0791c4a65a1a13edd00dca13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.167675 1767 consensus_queue.cc:260] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:13.168170 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 637) took 1931 ms. Trace:
I20251024 19:48:13.168267 1774 rpcz_store.cc:276] 1024 19:48:11.237074 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.237555 (+ 481us) service_pool.cc:225] Handling call
1024 19:48:12.524757 (+1287202us) ts_tablet_manager.cc:1151] Deleting tablet b6661b9e9a174849b6093d09cd8c1251
1024 19:48:12.524766 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:12.812403 (+287637us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:12.979317 (+166914us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:13.168151 (+188834us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1286883}
I20251024 19:48:13.168426 1767 ts_tablet_manager.cc:1434] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251024 19:48:13.168666 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 640) took 1910 ms. Trace:
I20251024 19:48:13.168746 1774 rpcz_store.cc:276] 1024 19:48:11.258071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.258128 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:13.168540 (+1910412us) ts_tablet_manager.cc:1151] Deleting tablet 46c1e2cef6cf4772b624060afb6fb117
1024 19:48:13.168549 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:13.168640 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1910212}
I20251024 19:48:13.168991 1767 tablet_bootstrap.cc:492] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:13.169059 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 641) took 1814 ms. Trace:
I20251024 19:48:13.169132 1774 rpcz_store.cc:276] 1024 19:48:11.354214 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.354276 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:13.168960 (+1814684us) ts_tablet_manager.cc:1151] Deleting tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9
1024 19:48:13.168968 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:13.169043 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1814413}
I20251024 19:48:13.169379 1774 tablet_replica.cc:333] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:13.114754 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 8eec3aa8f33746c09b63c8f41168eab1 (DEFAULT_TABLE table=default.Table_e3d3cfc2bb044e84acb9624615065375 [id=bd92e84f88c543918f6ad9c91b31d1f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.115463 1333 tablet_service.cc:1505] Processing CreateTablet for tablet f6df45dfab124358b8669bb4f01a6dd5 (DEFAULT_TABLE table=default.Table_33fc19a822a142afb669502687f7980c [id=932ebe916fab4be0b5f2697f11b56105]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.116328 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 1f5dfcab4dd242c580e2f3214a72837e (DEFAULT_TABLE table=default.Table_9ea29ee3999043faa320834c6c8f12b7 [id=a7ebc3b1f6004cde867cd808f13037be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.170461 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet fce7de9523fb4878a58aa8aef0c6b182 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:13.171430 1767 tablet_bootstrap.cc:654] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:13.172670 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 37163451258c436ab1f874dd0d66d393 (DEFAULT_TABLE table=default.Table_922f48f987904120a61a2eba3d270993 [id=1958f311b2f549888ecce9fb451136a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.173008 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37163451258c436ab1f874dd0d66d393. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.119387 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet fdfa5eea3b674a3389a156d1c72204df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.120466 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 64325d89db714554866e74922d103e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.126052 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 5b4f345b3b974681902f37b3975ea618 (DEFAULT_TABLE table=default.Table_acc74be61adc4422bd3935ae750cf797 [id=51219be242af44fa8fbbe079eb9eb7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.127866 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet d731403fbe0b4f92ac5899e13ea7bb19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.129093 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet 769f576810ab49368eb0fc7837b7da58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.131562 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 20e56d70049542f881f9490e4205dce3 (DEFAULT_TABLE table=default.Table_4da9eaf31f3d4d80870061b13b95cad5 [id=e82b4c1655f643a2b131a896abb41763]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.131834 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet fce7de9523fb4878a58aa8aef0c6b182 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.132140 1339 tablet_service.cc:1505] Processing CreateTablet for tablet ce593bda7c8547bb9cf2b30ca6820776 (DEFAULT_TABLE table=default.Table_0e854ff9b0b84dcbbdea3ec2c97a7aef [id=146333cfa7a04fe89852bd0b82c37fa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.133596 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet d9bf08ccf7054fedb9efcbb00354c71a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.140575 1768 raft_consensus.cc:359] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:13.065887 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03cdbd1b62b741249ad78562a7edfd22" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:13.183964 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 1da699a314ce4984b7566145ffb74ef1 (DEFAULT_TABLE table=default.Table_475f0aa37b7b4cff876223e3d8a0514c [id=d4dbe7cda31e4901b0e1aed96d5670ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.185019 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da699a314ce4984b7566145ffb74ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.090194 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ffd446c269448d38cc8d987a53aecf9" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:13.192175 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 20e56d70049542f881f9490e4205dce3 (DEFAULT_TABLE table=default.Table_4da9eaf31f3d4d80870061b13b95cad5 [id=e82b4c1655f643a2b131a896abb41763]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.192611 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20e56d70049542f881f9490e4205dce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.195716 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 583d0ef19aad4a80a8da15f7abc481c2 (DEFAULT_TABLE table=default.Table_8488a13f677347b591b4efde0761451b [id=5d07be972d4743979b659603ab7c2da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.196466 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 583d0ef19aad4a80a8da15f7abc481c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.203172 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 9275741737ba44e69b18a9419a10d6a6 (DEFAULT_TABLE table=default.Table_092242c51bbb4695947456a8ea5513dc [id=65830805c2cc488fa128ce367145ec2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.203565 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9275741737ba44e69b18a9419a10d6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.212213 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 104176d304384d1b99b051ae5744eec8 (DEFAULT_TABLE table=default.Table_71699a436b8045e6a4f9abdcb32351da [id=1da6e53aaefb4170977ada79d336a35c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.212703 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104176d304384d1b99b051ae5744eec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.213752 1767 tablet_bootstrap.cc:492] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:13.213981 1767 ts_tablet_manager.cc:1403] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20251024 19:48:13.215866 1767 raft_consensus.cc:359] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:13.216212 1767 raft_consensus.cc:385] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:13.216346 1767 raft_consensus.cc:740] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:13.217057 1767 consensus_queue.cc:260] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:13.218612 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4
W20251024 19:48:13.211454 1586 leader_election.cc:343] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:13.221716 1767 ts_tablet_manager.cc:1434] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20251024 19:48:13.222919 1767 tablet_bootstrap.cc:492] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:13.224908 1767 tablet_bootstrap.cc:654] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:13.228021 1628 tablet_service.cc:1505] Processing CreateTablet for tablet b22d207fbc234913b41a781c8836419e (DEFAULT_TABLE table=default.Table_135cfe439b8e4e44a4317938ff165eaf [id=97bd5f6c47e647d09c8e9c509a585891]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.228471 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b22d207fbc234913b41a781c8836419e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.221710 1586 leader_election.cc:304] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:13.239493 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 830c8469331940fc85aaad58c9085e35 (DEFAULT_TABLE table=default.Table_2df2e83bbab54dcabedf78455d66e990 [id=dcf98544c5bf49d2befbf5bfc16119ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.240058 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 830c8469331940fc85aaad58c9085e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.170130 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eec3aa8f33746c09b63c8f41168eab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.242470 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20e56d70049542f881f9490e4205dce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.243166 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6df45dfab124358b8669bb4f01a6dd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.174593 1768 raft_consensus.cc:385] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:13.245677 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b4f345b3b974681902f37b3975ea618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.245707 1768 raft_consensus.cc:740] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:13.245877 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce593bda7c8547bb9cf2b30ca6820776. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.245925 1768 consensus_queue.cc:260] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:13.246472 1768 ts_tablet_manager.cc:1434] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.107s user 0.000s sys 0.002s
I20251024 19:48:13.246760 1768 tablet_bootstrap.cc:492] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:13.182044 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 830c8469331940fc85aaad58c9085e35 (DEFAULT_TABLE table=default.Table_2df2e83bbab54dcabedf78455d66e990 [id=dcf98544c5bf49d2befbf5bfc16119ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.182420 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet b22b58404b8b441ba97ef536885a66e0 with delete_type TABLET_DATA_DELETED (replaced by tablet f37a01dc3d5f458d81605f2c41fa27dd) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.182464 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet ad055dc3387b414bb55d09b9683ba17e with delete_type TABLET_DATA_DELETED (replaced by tablet e83527f7ff5c4e248f344a84515a5d6a) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.249794 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 830c8469331940fc85aaad58c9085e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.249975 1483 tablet_service.cc:1505] Processing CreateTablet for tablet ce593bda7c8547bb9cf2b30ca6820776 (DEFAULT_TABLE table=default.Table_0e854ff9b0b84dcbbdea3ec2c97a7aef [id=146333cfa7a04fe89852bd0b82c37fa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.250486 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce593bda7c8547bb9cf2b30ca6820776. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.258926 1768 tablet_bootstrap.cc:654] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:13.263315 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f5dfcab4dd242c580e2f3214a72837e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.270514 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cef80e4d14bb4e719bceebf43041d079" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:13.270942 2569 raft_consensus.cc:2749] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:13.271127 1447 leader_election.cc:343] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:13.271149 2569 raft_consensus.cc:493] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:13.271268 2569 raft_consensus.cc:515] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:13.271428 1447 leader_election.cc:343] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:13.271693 1447 leader_election.cc:343] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:13.271965 1447 leader_election.cc:343] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:13.275888 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet b22b58404b8b441ba97ef536885a66e0 with delete_type TABLET_DATA_DELETED (replaced by tablet f37a01dc3d5f458d81605f2c41fa27dd) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:13.270978 2810 consensus_queue.cc:1048] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 0, Time since last communication: 0.007s
I20251024 19:48:13.286391 1768 tablet_bootstrap.cc:492] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:13.286727 1768 ts_tablet_manager.cc:1403] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.009s
I20251024 19:48:13.288825 1768 raft_consensus.cc:359] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:13.289191 1768 raft_consensus.cc:385] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:13.289291 1768 raft_consensus.cc:740] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:13.289561 1768 consensus_queue.cc:260] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:13.290591 1768 ts_tablet_manager.cc:1434] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251024 19:48:13.290987 1768 tablet_bootstrap.cc:492] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:13.291416 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet ad055dc3387b414bb55d09b9683ba17e with delete_type TABLET_DATA_DELETED (replaced by tablet e83527f7ff5c4e248f344a84515a5d6a) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:13.293736 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 177728us
I20251024 19:48:13.293907 1768 tablet_bootstrap.cc:654] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:13.294175 1766 raft_consensus.cc:359] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.294512 1766 raft_consensus.cc:385] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:13.294631 1766 raft_consensus.cc:740] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:13.294930 1766 consensus_queue.cc:260] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.295863 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 5b96fc2309274048bed7a35e4bf4bae2 (table default.Table_dc366a1f70d84d4f8276c1872b9fa0f7 [id=0f190205e9b347f6906d6f86b0624681]) successfully deleted
I20251024 19:48:13.299634 1766 ts_tablet_manager.cc:1434] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.198s user 0.002s sys 0.000s
I20251024 19:48:13.299955 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet b6661b9e9a174849b6093d09cd8c1251 (table default.Table_7dbe5740202642f1be406c4897a72541 [id=df94703d2472430387b81c028130d628]) successfully deleted
I20251024 19:48:13.300108 1766 tablet_bootstrap.cc:492] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:13.302049 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 184066us
I20251024 19:48:13.303110 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cef80e4d14bb4e719bceebf43041d079" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:13.303412 1648 raft_consensus.cc:3060] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:13.304227 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 46c1e2cef6cf4772b624060afb6fb117 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46c1e2cef6cf4772b624060afb6fb117 already in progress: creating tablet
W20251024 19:48:13.308651 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 already in progress: creating tablet
I20251024 19:48:13.310403 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 0a12e415590a4c71aab4ad7d30412c83 (table default.Table_1a079b981b694617b3baf9897dfb37ae [id=18ec057e920e4c21ae8cbe298058b6f3]) successfully deleted
I20251024 19:48:13.169561 1774 raft_consensus.cc:2243] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:13.316777 1774 raft_consensus.cc:2272] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:13.317369 1774 ts_tablet_manager.cc:1916] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:13.322283 1766 tablet_bootstrap.cc:654] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:13.327265 1628 tablet_service.cc:1505] Processing CreateTablet for tablet d0cbe0a3436a498a84c892ac156798ed (DEFAULT_TABLE table=default.Table_4dca2888019448ca97c68a8c6dd8396a [id=9da07f2aef964c9a838808d27f211636]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:13.327445 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3324755a5a184a609fae5f7a25d58e4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3324755a5a184a609fae5f7a25d58e4c already in progress: creating tablet
I20251024 19:48:13.327769 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cbe0a3436a498a84c892ac156798ed. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:13.328007 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 780442b310e6409498e6bbe4256280a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 780442b310e6409498e6bbe4256280a4 already in progress: creating tablet
W20251024 19:48:13.328332 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 65424b00ab9c454f9584048caba83e6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65424b00ab9c454f9584048caba83e6a already in progress: creating tablet
W20251024 19:48:13.328883 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7eb62791100f477784ab315923b6ceb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7eb62791100f477784ab315923b6ceb6 already in progress: creating tablet
W20251024 19:48:13.329270 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 83b91a5d764449d5b82ef6d1ee0360f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83b91a5d764449d5b82ef6d1ee0360f5 already in progress: creating tablet
I20251024 19:48:13.330889 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.330926 1775 ts_tablet_manager.cc:1929] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:13.331022 1775 log.cc:1199] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/aebf76d4a66947ed9756643740418798
I20251024 19:48:13.331339 1775 ts_tablet_manager.cc:1950] T aebf76d4a66947ed9756643740418798 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:13.332188 1766 tablet_bootstrap.cc:492] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:13.332396 1766 ts_tablet_manager.cc:1403] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20251024 19:48:13.332608 1344 tablet_service.cc:1505] Processing CreateTablet for tablet b74415cf7c7e47a696b454a5d44d8017 (DEFAULT_TABLE table=default.Table_f5ee199153824340833717cac0a3cea0 [id=41569aa93ac34c758747a9abc5483d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.333408 1766 raft_consensus.cc:359] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.333850 1766 raft_consensus.cc:385] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:13.333989 1766 raft_consensus.cc:740] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:13.333918 1488 tablet_service.cc:1505] Processing CreateTablet for tablet b74415cf7c7e47a696b454a5d44d8017 (DEFAULT_TABLE table=default.Table_f5ee199153824340833717cac0a3cea0 [id=41569aa93ac34c758747a9abc5483d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.334272 1766 consensus_queue.cc:260] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.334647 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74415cf7c7e47a696b454a5d44d8017. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.334909 1766 ts_tablet_manager.cc:1434] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251024 19:48:13.335237 1766 tablet_bootstrap.cc:492] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:13.335657 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 027e12c7961247a4a1664b85e4ddae4c (DEFAULT_TABLE table=default.Table_6b101c11690c47319e38598ef667655b [id=c071f24459334271be5e16b90bee2a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.336081 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74415cf7c7e47a696b454a5d44d8017. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.336148 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 027e12c7961247a4a1664b85e4ddae4c (DEFAULT_TABLE table=default.Table_6b101c11690c47319e38598ef667655b [id=c071f24459334271be5e16b90bee2a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.336558 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 027e12c7961247a4a1664b85e4ddae4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.339586 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 027e12c7961247a4a1664b85e4ddae4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.343379 1366 raft_consensus.cc:3060] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:13.343464 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 132040us
W20251024 19:48:13.343729 1586 consensus_peers.cc:597] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:13.343994 2569 leader_election.cc:290] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:13.346387 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 520060a9adca4bb0b8616bcc006dce52 (DEFAULT_TABLE table=default.Table_9e7fa6339ee24bab9d78fd7c318ed78d [id=b03b545fcd2f47168d4309e5a6c5ea2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.346477 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 3c608ac55e9e42bc9c9b27beec717e30 (DEFAULT_TABLE table=default.Table_d61970c7cdd9475d9d6da8dc5f80a232 [id=bf39bf58617e4d36a1b1de926062b633]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.346814 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c608ac55e9e42bc9c9b27beec717e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.347412 2567 consensus_queue.cc:1048] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251024 19:48:13.349460 1474 tablet_service.cc:1505] Processing CreateTablet for tablet f900b961f12e4e469ac3f440de0bf78e (DEFAULT_TABLE table=default.Table_8638421b2d894a4dbf4a9f06afc4c595 [id=5c65856731f2409abd549da6fa103672]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.349908 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f900b961f12e4e469ac3f440de0bf78e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.350651 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_92ca5409f01446e494834187689f234d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.351202 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92ca5409f01446e494834187689f234d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.356882 1620 tablet_service.cc:1505] Processing CreateTablet for tablet c694a372afc94f89972c972793c5e901 (DEFAULT_TABLE table=default.Table_43dbb020b49b44cfa627ef85edba938c [id=7d5fa7422f654970b1eee19e5f6d4360]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.357158 1627 tablet_service.cc:1505] Processing CreateTablet for tablet f709db684b6c49cbaeef3bf8d76f5828 (DEFAULT_TABLE table=default.Table_5bfbf5d7f5764b96a95bf3ac0e0aba69 [id=9a49120c5b654676a63cd3966eaf31a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.357340 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 520060a9adca4bb0b8616bcc006dce52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.357529 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f709db684b6c49cbaeef3bf8d76f5828. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.359444 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e03c488d0f264684b0897a81a04d7466" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:13.359637 1510 raft_consensus.cc:2468] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:13.360344 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 6e80dfddd21b41e98ebd226e0219b519 (DEFAULT_TABLE table=default.Table_5b54ca8cd83f49188876e061f70d629e [id=d587a8d80edb400daefac34b4175e4dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.360725 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e80dfddd21b41e98ebd226e0219b519. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.361487 1586 leader_election.cc:304] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:13.361886 2569 raft_consensus.cc:2804] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:13.361974 2569 raft_consensus.cc:493] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:13.362044 2569 raft_consensus.cc:3060] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:13.362648 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5114b358c69441539cfc49077ea63f9e" } modify_external_catalogs: true
I20251024 19:48:13.362829 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5114b358c69441539cfc49077ea63f9e" } modify_external_catalogs: true
I20251024 19:48:13.363623 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d27cfa82d3b04992b5e7d5d77c884000"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.364110 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d27cfa82d3b04992b5e7d5d77c884000 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.374245 1511 raft_consensus.cc:3060] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:13.376350 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 60fbfb1620934034af2acaa39677554f (DEFAULT_TABLE table=default.Table_df275b460e8245c8bc231bab1fb44630 [id=08c466bccc854f82a9da4212e7ce813f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.376775 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60fbfb1620934034af2acaa39677554f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.363574 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ac33dd71eb504d6a8d92fef51601721e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.378502 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac33dd71eb504d6a8d92fef51601721e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.378609 1766 tablet_bootstrap.cc:654] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:13.381131 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c694a372afc94f89972c972793c5e901. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.386135 2640 consensus_queue.cc:1048] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:13.348147 1337 tablet_service.cc:1505] Processing CreateTablet for tablet f900b961f12e4e469ac3f440de0bf78e (DEFAULT_TABLE table=default.Table_8638421b2d894a4dbf4a9f06afc4c595 [id=5c65856731f2409abd549da6fa103672]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.389542 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f900b961f12e4e469ac3f440de0bf78e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.396428 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:13.399442 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_96fd924db35c4970944db55c46bace57" } new_table_name: "default.Table_83edb13b9385493789798a9cbc27dcdf" modify_external_catalogs: true
I20251024 19:48:13.401317 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a8e5ad4c50ba487c90b7e49a84c41357" } new_table_name: "default.Table_630eb735037f4337989ff580a3a6eee4" modify_external_catalogs: true
I20251024 19:48:13.398706 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 3c608ac55e9e42bc9c9b27beec717e30 (DEFAULT_TABLE table=default.Table_d61970c7cdd9475d9d6da8dc5f80a232 [id=bf39bf58617e4d36a1b1de926062b633]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.403160 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c608ac55e9e42bc9c9b27beec717e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.403524 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2ffcc8a68d3f46d381d6c565f890c6ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.404111 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ffcc8a68d3f46d381d6c565f890c6ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.405544 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_485cf0780a4c4dfe809deb6fb31a4671"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.406102 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_485cf0780a4c4dfe809deb6fb31a4671 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.406047 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_98740f996f06427dbd2383bf1c19bc1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.406494 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98740f996f06427dbd2383bf1c19bc1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.407847 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f16963cab00043a4887d93a6326a1d9c" } new_table_name: "default.Table_738f1cac13cb4cff9e3d7e3a36be4af1" modify_external_catalogs: true
I20251024 19:48:13.408245 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1865562320ef412799f93a15c66ade0c" } modify_external_catalogs: true
I20251024 19:48:13.408414 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1865562320ef412799f93a15c66ade0c" } modify_external_catalogs: true
I20251024 19:48:13.409932 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_082186ad39f14242b8dbcf092a97ae7d" } modify_external_catalogs: true
I20251024 19:48:13.410097 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_082186ad39f14242b8dbcf092a97ae7d" } modify_external_catalogs: true
I20251024 19:48:13.410084 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 10a19c15fc0d4e3e82373c51c59d18fd (DEFAULT_TABLE table=default.Table_281f4fc881c345e7b86cdc489422126b [id=e0c418ba3dd54e889b0bdbd0949bec1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.410492 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10a19c15fc0d4e3e82373c51c59d18fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.404551 1766 tablet_bootstrap.cc:492] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:13.418931 1766 ts_tablet_manager.cc:1403] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.001s
I20251024 19:48:13.420464 1766 raft_consensus.cc:359] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.423874 1766 raft_consensus.cc:385] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:13.424326 1766 raft_consensus.cc:740] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:13.425025 1766 consensus_queue.cc:260] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.426759 1766 ts_tablet_manager.cc:1434] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251024 19:48:13.427150 1766 tablet_bootstrap.cc:492] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:13.429000 1766 tablet_bootstrap.cc:654] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:13.354940 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet b79801594e95434cbb8fe7b75b14602e with delete_type TABLET_DATA_DELETED (replaced by tablet 3e6f8764e9ae472b9ac35a032182aed1) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.364051 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 6e80dfddd21b41e98ebd226e0219b519 (DEFAULT_TABLE table=default.Table_5b54ca8cd83f49188876e061f70d629e [id=d587a8d80edb400daefac34b4175e4dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.436296 1648 raft_consensus.cc:2468] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
W20251024 19:48:13.448422 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b2992a738b440d798f5e61cc45ec088 consensus state has no leader
W20251024 19:48:13.449625 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c8147e8a6e0141669f109b41a8319398 consensus state has no leader
I20251024 19:48:13.450089 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 506) took 6719 ms. Trace:
I20251024 19:48:13.450273 1775 rpcz_store.cc:276] 1024 19:48:06.730747 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.730813 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:12.854457 (+6123644us) ts_tablet_manager.cc:1151] Deleting tablet aebf76d4a66947ed9756643740418798
1024 19:48:12.854470 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:13.056899 (+202429us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:13.330821 (+273922us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:13.450062 (+119241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6123420}
I20251024 19:48:13.451175 1775 tablet_replica.cc:333] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:13.451463 1775 raft_consensus.cc:2243] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:13.451735 1775 raft_consensus.cc:2272] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:13.452488 1775 ts_tablet_manager.cc:1916] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:13.453369 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e80dfddd21b41e98ebd226e0219b519. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.456100 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet aebf76d4a66947ed9756643740418798 (table default.Table_d09f149249c44c50adcb71321c133f2d [id=665389c7bde54b7786edabcb91a9b33b]) successfully deleted
I20251024 19:48:13.472949 1191 catalog_manager.cc:5649] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:13.473565 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 03cdbd1b62b741249ad78562a7edfd22 (table default.Table_42c3a4a03db14d0684721ce6af6960c2 [id=ca742ae585e0406abf43193a23aa9731]). Expected version 2 got 0
I20251024 19:48:13.474493 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 635a132735704cfda69bf7c8deaea81d (table default.Table_933a6697624149369ab3ec28462b466a [id=93a61862707b4a7ba8a0ff349704dc2b]). Expected version 1 got 0
I20251024 19:48:13.475952 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 2764689179504e079924fde7e425dbcd (DEFAULT_TABLE table=default.Table_e63d3e693e1148a7b8d2b5731b1027bc [id=5fc5a169166b436ba9343a652a650b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.476519 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2764689179504e079924fde7e425dbcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.484090 1333 tablet_service.cc:1505] Processing CreateTablet for tablet e0228d5b788e4c42bf4ecce876ecc3f4 (DEFAULT_TABLE table=default.Table_5afc602c5d154cf48565fa17d88d7c68 [id=cc27a6fdb94f4f6d876a9a627dcbe7be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.484593 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0228d5b788e4c42bf4ecce876ecc3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.494650 1766 tablet_bootstrap.cc:492] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:13.507071 1766 ts_tablet_manager.cc:1403] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.080s user 0.000s sys 0.010s
I20251024 19:48:13.499336 2796 raft_consensus.cc:493] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:13.508096 2796 raft_consensus.cc:515] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.509246 2796 leader_election.cc:290] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:13.511796 1766 raft_consensus.cc:359] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.513684 1766 raft_consensus.cc:385] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:13.513809 1766 raft_consensus.cc:740] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:13.503026 2569 raft_consensus.cc:515] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.514321 1766 consensus_queue.cc:260] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.515192 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 22fa56d227f54eaf9f58ca116e730931 (DEFAULT_TABLE table=default.Table_aa1ad4ed1e214bb890edff3ebe855022 [id=7a58cfd8391e47439763a78cd17af696]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.515270 1766 ts_tablet_manager.cc:1434] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251024 19:48:13.515558 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fa56d227f54eaf9f58ca116e730931. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.516577 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 2c9ff67824254c04bd7d024cbcc6c080 (DEFAULT_TABLE table=default.Table_b1bf5228ae5f485abeb61ea7d512b30d [id=a38b28fa71b14c41b090b6c312ada1bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.516970 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c9ff67824254c04bd7d024cbcc6c080. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.520931 1613 tablet_service.cc:1505] Processing CreateTablet for tablet fd6aa122a7504daba11aea3e7f507389 (DEFAULT_TABLE table=default.Table_3253b4bae4f14474b0c319b84e6b3130 [id=b0eaaa50b6174d188711550594d42ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.521298 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd6aa122a7504daba11aea3e7f507389. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.522763 1488 tablet_service.cc:1505] Processing CreateTablet for tablet e0228d5b788e4c42bf4ecce876ecc3f4 (DEFAULT_TABLE table=default.Table_5afc602c5d154cf48565fa17d88d7c68 [id=cc27a6fdb94f4f6d876a9a627dcbe7be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.522864 1477 tablet_service.cc:1505] Processing CreateTablet for tablet e83527f7ff5c4e248f344a84515a5d6a (DEFAULT_TABLE table=default.Table_cde3670bf0d04dc0b2049193bcee10d4 [id=c3de6937e93a48dcb88cabee59a09d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.522863 1621 tablet_service.cc:1505] Processing CreateTablet for tablet dcd06be0791c4a65a1a13edd00dca13b (DEFAULT_TABLE table=default.Table_44475cd37f524126a9ac72ec03a5712c [id=538e81755a22418ab7c8833124cd680a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.523134 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0228d5b788e4c42bf4ecce876ecc3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.523180 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcd06be0791c4a65a1a13edd00dca13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.523170 1483 tablet_service.cc:1505] Processing CreateTablet for tablet f37a01dc3d5f458d81605f2c41fa27dd (DEFAULT_TABLE table=default.Table_f16963cab00043a4887d93a6326a1d9c [id=bdfe9c0f34ac41eebb843bd17fa9635c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.524901 1766 tablet_bootstrap.cc:492] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:13.526140 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f37a01dc3d5f458d81605f2c41fa27dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.526676 1766 tablet_bootstrap.cc:654] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:13.528637 1774 ts_tablet_manager.cc:1929] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:13.528776 1774 log.cc:1199] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/2037fcca87b94b5fb31639b4d07307b0
I20251024 19:48:13.529102 1774 ts_tablet_manager.cc:1950] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:13.529433 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 642) took 2152 ms. Trace:
I20251024 19:48:13.530050 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e83527f7ff5c4e248f344a84515a5d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.530151 1616 tablet_service.cc:1505] Processing CreateTablet for tablet c8147e8a6e0141669f109b41a8319398 (DEFAULT_TABLE table=default.Table_96fd924db35c4970944db55c46bace57 [id=189e9845121c4f22a7a1f16a581fadaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.530714 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8147e8a6e0141669f109b41a8319398. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.531088 1625 tablet_service.cc:1505] Processing CreateTablet for tablet b383a83425654723bd1d9c3a39b187be (DEFAULT_TABLE table=default.Table_09f15fac93da45ab80457bd1efc49464 [id=47ed56aa60974fef835ce3cc8d3a713d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.531489 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d04a9b54bd6446d8deb64292cef672d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.531688 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet 607ae453221d422e9d07882830c02275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.531972 1629 tablet_service.cc:1505] Processing CreateTablet for tablet fc35c0a2b36c45ffa2fe4ad00abccfd1 (DEFAULT_TABLE table=default.Table_35b42b8f59b042ba9161297fca0d9bd3 [id=ddd649b907b64761a8ad29ba023d9a32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.532413 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 63518b583ac94fa6abde420e176ec8ba (DEFAULT_TABLE table=default.Table_ce9762cc2ae44e2dbf2404da0554444c [id=47ae0764c00249b78055e4a5beb29248]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.533480 2569 leader_election.cc:290] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:13.549010 2796 raft_consensus.cc:493] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:13.549371 2796 raft_consensus.cc:515] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.550658 2796 leader_election.cc:290] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:13.550995 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f75802f81eff41248b27d50a04c4efbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.551913 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f75802f81eff41248b27d50a04c4efbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.559546 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e63d3e693e1148a7b8d2b5731b1027bc" } modify_external_catalogs: true
I20251024 19:48:13.559854 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e63d3e693e1148a7b8d2b5731b1027bc" } modify_external_catalogs: true
I20251024 19:48:13.563319 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 9f791083b4964ff9b0f5fc575478892c (DEFAULT_TABLE table=default.Table_867f1b66094349fa8d81af1e89bebb06 [id=3959b7b1ee654e149138eb210c3455a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.565421 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1ece0128472c4ef2a4e5f261fe73b3a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.566030 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ece0128472c4ef2a4e5f261fe73b3a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.579432 1611 tablet_service.cc:1505] Processing CreateTablet for tablet ec11453f49054f9dbbe620374e59a6ce (DEFAULT_TABLE table=default.Table_abc327da99a44b9d9f01f12c54b1ede2 [id=d0ed1317be5c4d6d915e0ff881de5acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.591328 2781 raft_consensus.cc:493] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:13.591542 2781 raft_consensus.cc:515] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:13.591820 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f37a01dc3d5f458d81605f2c41fa27dd consensus state has no leader
I20251024 19:48:13.592379 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_56bed2d1fdc64b03a33e778185a2f371" } new_table_name: "default.Table_16ba8561c0c349f0937e04f2ab1ed21a" modify_external_catalogs: true
I20251024 19:48:13.594098 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2346c03802a245a2af8c086b626d660d
I20251024 19:48:13.595088 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f518918f7bfe409fae8f42ac3d059872"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.595713 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f518918f7bfe409fae8f42ac3d059872 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.597864 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 2346c03802a245a2af8c086b626d660d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:13.597795 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c7859afca8534349935391ebab856798"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.598299 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7859afca8534349935391ebab856798 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.599978 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_29c99dcba4ab4430aa0b7cb3c70621e9" } new_table_name: "default.Table_539d0fd237684238a18e3abd9599b632" modify_external_catalogs: true
I20251024 19:48:13.601397 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 4243f11bf2b34b30b10766c19e2c6409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.604486 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec11453f49054f9dbbe620374e59a6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.606065 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b383a83425654723bd1d9c3a39b187be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.606503 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc35c0a2b36c45ffa2fe4ad00abccfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.606920 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63518b583ac94fa6abde420e176ec8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.611018 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 4243f11bf2b34b30b10766c19e2c6409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:13.611292 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet cef80e4d14bb4e719bceebf43041d079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:13.615324 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_83b4409228694c939a4a734c3b6bdd76" } modify_external_catalogs: true
I20251024 19:48:13.616760 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_83b4409228694c939a4a734c3b6bdd76" } modify_external_catalogs: true
I20251024 19:48:13.624378 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 73278db2097d4001a538d62dacb97c9b
W20251024 19:48:13.626717 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet c8147e8a6e0141669f109b41a8319398 (table default.Table_83edb13b9385493789798a9cbc27dcdf [id=189e9845121c4f22a7a1f16a581fadaa], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c8147e8a6e0141669f109b41a8319398 consensus state has no leader
W20251024 19:48:13.630411 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 175458us
I20251024 19:48:13.634280 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 73278db2097d4001a538d62dacb97c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:13.634590 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e03c488d0f264684b0897a81a04d7466" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:13.634790 1511 raft_consensus.cc:3060] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:13.634931 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f791083b4964ff9b0f5fc575478892c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.636250 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 5b4f345b3b974681902f37b3975ea618 (DEFAULT_TABLE table=default.Table_acc74be61adc4422bd3935ae750cf797 [id=51219be242af44fa8fbbe079eb9eb7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.636617 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b4f345b3b974681902f37b3975ea618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.637712 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 187bc968c2b049028dd25dfb0264c0aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.637996 1511 raft_consensus.cc:2468] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:13.638279 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 1f5dfcab4dd242c580e2f3214a72837e (DEFAULT_TABLE table=default.Table_9ea29ee3999043faa320834c6c8f12b7 [id=a7ebc3b1f6004cde867cd808f13037be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.638414 1586 leader_election.cc:304] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:13.638711 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f5dfcab4dd242c580e2f3214a72837e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.638695 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9377725db312439e8a42dbd46da84a9d
I20251024 19:48:13.639501 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 8eec3aa8f33746c09b63c8f41168eab1 (DEFAULT_TABLE table=default.Table_e3d3cfc2bb044e84acb9624615065375 [id=bd92e84f88c543918f6ad9c91b31d1f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.640569 1614 tablet_service.cc:1505] Processing CreateTablet for tablet f6df45dfab124358b8669bb4f01a6dd5 (DEFAULT_TABLE table=default.Table_33fc19a822a142afb669502687f7980c [id=932ebe916fab4be0b5f2697f11b56105]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.640856 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 9377725db312439e8a42dbd46da84a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:13.643225 1192 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet e29abfa3bc8944aea7d1acd03e3eb897 (table default.Table_55339e02660843c3a8536a4b7a06831e [id=ba30503223dd4979824ec9c1a50a8ae2]). Expected version 1 got 0
I20251024 19:48:13.643576 1627 tablet_service.cc:1552] Processing DeleteTablet for tablet 64325d89db714554866e74922d103e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.644479 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f145f64b26134e0e9fa7c293d5d7842f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.645247 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f145f64b26134e0e9fa7c293d5d7842f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.647548 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_80225fc48e434efc981690a21c61cd00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.648094 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80225fc48e434efc981690a21c61cd00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.650823 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e03c488d0f264684b0897a81a04d7466" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:13.657711 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce8c179e98944df0b510a0e8daf3920f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:13.658653 1344 tablet_service.cc:1505] Processing CreateTablet for tablet f37a01dc3d5f458d81605f2c41fa27dd (DEFAULT_TABLE table=default.Table_f16963cab00043a4887d93a6326a1d9c [id=bdfe9c0f34ac41eebb843bd17fa9635c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.659214 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f37a01dc3d5f458d81605f2c41fa27dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.659471 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e03c488d0f264684b0897a81a04d7466" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:13.659219 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 2346c03802a245a2af8c086b626d660d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:13.662490 1586 leader_election.cc:343] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:13.662959 1586 leader_election.cc:343] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:13.500670 1444 consensus_peers.cc:597] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdfa5eea3b674a3389a156d1c72204df. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:13.665375 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 164716us
I20251024 19:48:13.669169 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2764689179504e079924fde7e425dbcd
I20251024 19:48:13.670161 1776 ts_tablet_manager.cc:1929] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:13.670054 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a157b3f500ac43c285d1b0c80f0bd88b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:13.670198 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 2764689179504e079924fde7e425dbcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:13.670271 1776 log.cc:1199] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/d9bf08ccf7054fedb9efcbb00354c71a
I20251024 19:48:13.670629 1776 ts_tablet_manager.cc:1950] T d9bf08ccf7054fedb9efcbb00354c71a P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
W20251024 19:48:13.670658 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a157b3f500ac43c285d1b0c80f0bd88b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.672490 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_587f4b6c35e94fbca64f79a03aecf9c1" } new_table_name: "default.Table_25169e6ef41048cab53f651e950e992f" modify_external_catalogs: true
I20251024 19:48:13.682806 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6df45dfab124358b8669bb4f01a6dd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.687836 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 424bce2f496b4a4fa74e57794f6a11eb (DEFAULT_TABLE table=default.Table_a478a67f294e4fdcba8d2f2b421d26fd [id=2e0492e797864318bb9bfa9ffce2be31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.688299 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 424bce2f496b4a4fa74e57794f6a11eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.688623 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eec3aa8f33746c09b63c8f41168eab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.689133 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 95e11ed28aab4e809e8fc1615f21744c (DEFAULT_TABLE table=default.Table_babcfed7fd5b4b6ebf17c81871843326 [id=288378453f244ce3aae05bbd428e0ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.689486 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95e11ed28aab4e809e8fc1615f21744c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.692441 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet dc14b0cdb7f342ac956a7857ab3f35d7 (table default.Table_be61dd09c4c94f568d723c4bace3a038 [id=66be3fc672ea4a9e9fb5dde3ae5e5b62]) successfully deleted
W20251024 19:48:13.692791 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 already in progress: creating tablet
I20251024 19:48:13.698047 2561 raft_consensus.cc:493] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:13.698302 2561 raft_consensus.cc:515] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.699384 2561 leader_election.cc:290] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:13.701958 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2cbeaac19e834ecf98e5b8afc952b9da consensus state has no leader
I20251024 19:48:13.706398 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_25c67815f6fc45c780e1a9fd8ae678f9" } new_table_name: "default.Table_9b49c126d44b4afeb3d1122999d2b8da" modify_external_catalogs: true
I20251024 19:48:13.709823 2781 leader_election.cc:290] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:13.717800 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 769f576810ab49368eb0fc7837b7da58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.718093 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet d731403fbe0b4f92ac5899e13ea7bb19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.727898 2781 raft_consensus.cc:493] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:13.728066 2781 raft_consensus.cc:515] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:13.728729 2781 leader_election.cc:290] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:13.731120 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 73278db2097d4001a538d62dacb97c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.743021 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_542a694472c54365a9cc9ae4d484013e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.744354 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_542a694472c54365a9cc9ae4d484013e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.746588 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 830c8469331940fc85aaad58c9085e35 (DEFAULT_TABLE table=default.Table_2df2e83bbab54dcabedf78455d66e990 [id=dcf98544c5bf49d2befbf5bfc16119ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.747046 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 830c8469331940fc85aaad58c9085e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.747706 2569 raft_consensus.cc:2804] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:13.747820 2569 raft_consensus.cc:697] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:13.748579 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 9377725db312439e8a42dbd46da84a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.749002 1335 tablet_service.cc:1505] Processing CreateTablet for tablet e83527f7ff5c4e248f344a84515a5d6a (DEFAULT_TABLE table=default.Table_cde3670bf0d04dc0b2049193bcee10d4 [id=c3de6937e93a48dcb88cabee59a09d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.749972 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 2764689179504e079924fde7e425dbcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.751353 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9bf08ccf7054fedb9efcbb00354c71a" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:13.751614 1511 raft_consensus.cc:2393] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:13.751574 1623 tablet_service.cc:1505] Processing CreateTablet for tablet ce593bda7c8547bb9cf2b30ca6820776 (DEFAULT_TABLE table=default.Table_0e854ff9b0b84dcbbdea3ec2c97a7aef [id=146333cfa7a04fe89852bd0b82c37fa9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:13.751693 1447 leader_election.cc:343] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:13.750476 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e83527f7ff5c4e248f344a84515a5d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.752308 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b79801594e95434cbb8fe7b75b14602e" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:13.752636 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ffd446c269448d38cc8d987a53aecf9" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:13.752995 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07e75c7f35544bec82cb994dea5ee4ef" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:13.753742 1447 leader_election.cc:343] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:13.753988 1447 consensus_peers.cc:597] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:13.754143 1447 leader_election.cc:343] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:13.754321 1305 leader_election.cc:343] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: ad81d80697984ff1834a7d795e0f4b9b
I20251024 19:48:13.754455 1305 leader_election.cc:304] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:13.754724 2781 raft_consensus.cc:2749] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:13.754879 1305 leader_election.cc:343] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 59c7dbe9ff2c4869994c4f43d5676c30
I20251024 19:48:13.754972 1305 leader_election.cc:304] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:13.755187 2781 raft_consensus.cc:2749] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:13.755342 1305 leader_election.cc:343] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: fdfa5eea3b674a3389a156d1c72204df
I20251024 19:48:13.755455 1305 leader_election.cc:304] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:13.755626 2781 raft_consensus.cc:2749] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:13.756994 1305 leader_election.cc:343] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 2037fcca87b94b5fb31639b4d07307b0
I20251024 19:48:13.757086 1305 leader_election.cc:304] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
W20251024 19:48:13.757344 1305 raft_consensus.cc:2686] T 2037fcca87b94b5fb31639b4d07307b0 P 43bec2f03fc849099e254f6335e9d532: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251024 19:48:13.762214 1305 leader_election.cc:343] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: d9bf08ccf7054fedb9efcbb00354c71a
I20251024 19:48:13.763139 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet ad055dc3387b414bb55d09b9683ba17e with delete_type TABLET_DATA_DELETED (replaced by tablet e83527f7ff5c4e248f344a84515a5d6a) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.763334 1305 leader_election.cc:304] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:13.748103 2569 consensus_queue.cc:237] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:13.763779 1305 leader_election.cc:343] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: b79801594e95434cbb8fe7b75b14602e
I20251024 19:48:13.763876 1305 leader_election.cc:304] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:13.764305 2838 raft_consensus.cc:3060] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:13.764406 2781 raft_consensus.cc:2749] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:13.765180 2815 tablet.cc:1722] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251024 19:48:13.765787 1627 tablet_service.cc:1505] Processing CreateTablet for tablet 20e56d70049542f881f9490e4205dce3 (DEFAULT_TABLE table=default.Table_4da9eaf31f3d4d80870061b13b95cad5 [id=e82b4c1655f643a2b131a896abb41763]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.766407 1768 tablet_bootstrap.cc:492] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:13.766534 1768 ts_tablet_manager.cc:1403] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.476s user 0.017s sys 0.043s
I20251024 19:48:13.768222 1766 tablet_bootstrap.cc:492] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:13.769685 1766 ts_tablet_manager.cc:1403] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.245s user 0.019s sys 0.025s
I20251024 19:48:13.770906 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a8a6e3b550974b0e8267a1e5f3d769e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:13.771503 2838 raft_consensus.cc:2749] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:13.771553 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8a6e3b550974b0e8267a1e5f3d769e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:13.777132 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 157f84b56067415da404df30fc8a5755 consensus state has no leader
I20251024 19:48:13.778365 1768 raft_consensus.cc:359] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.778563 1768 raft_consensus.cc:385] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:13.778615 1768 raft_consensus.cc:740] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:13.777591 1766 raft_consensus.cc:359] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:13.779140 1766 raft_consensus.cc:385] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:13.779219 1766 raft_consensus.cc:740] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:13.779484 1766 consensus_queue.cc:260] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:13.780090 1766 ts_tablet_manager.cc:1434] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251024 19:48:13.780514 1768 consensus_queue.cc:260] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.780705 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_86e47213e04046e89c4ef6974b4b3630"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.781256 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86e47213e04046e89c4ef6974b4b3630 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.781370 1768 ts_tablet_manager.cc:1434] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251024 19:48:13.781687 1768 tablet_bootstrap.cc:492] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:13.783339 1768 tablet_bootstrap.cc:654] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:13.785356 1766 tablet_bootstrap.cc:492] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:13.787163 1766 tablet_bootstrap.cc:654] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:13.798977 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3fefd748cb624ce2bad33072fbbb3aec" } new_table_name: "default.Table_9147a98b3bf9483ea7b7a6f17130d4fc" modify_external_catalogs: true
I20251024 19:48:13.806156 2815 tablet.cc:1717] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532: Skipping requested alter to schema version 3, tablet already version 3
I20251024 19:48:13.809247 2808 raft_consensus.cc:493] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:13.809396 2808 raft_consensus.cc:515] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.810030 2808 leader_election.cc:290] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:13.810971 1447 leader_election.cc:343] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 2037fcca87b94b5fb31639b4d07307b0
I20251024 19:48:13.813444 1418 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:13.828892 2843 tablet.cc:1717] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e: Skipping requested alter to schema version 3, tablet already version 3
I20251024 19:48:13.830847 2843 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:46462 (request call id 567) took 5809 ms. Trace:
I20251024 19:48:13.833078 2843 rpcz_store.cc:276] 1024 19:48:08.021296 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.021533 (+ 237us) service_pool.cc:225] Handling call
1024 19:48:13.830827 (+5809294us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1024 19:48:08.042473 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1024 19:48:08.042592 (+ 119us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1024 19:48:08.496741 (+454149us) alter_schema_op.cc:66] Acquired schema lock
1024 19:48:08.496751 (+ 10us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1024 19:48:08.496796 (+ 45us) alter_schema_op.cc:128] START. Timestamp: 1761335288496753
1024 19:48:08.496799 (+ 3us) op_driver.cc:348] REPLICATION: starting
1024 19:48:08.496986 (+ 187us) log.cc:844] Serialized 173 byte log entry
1024 19:48:13.804875 (+5307889us) op_driver.cc:464] REPLICATION: finished
1024 19:48:13.828852 (+ 23977us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1024 19:48:13.830746 (+ 1894us) log.cc:844] Serialized 94 byte log entry
1024 19:48:13.830779 (+ 33us) alter_schema_op.cc:73] Released schema lock
1024 19:48:13.830785 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":23888,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":34729,"prepare.run_wall_time_us":455765,"raft.queue_time_us":43,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":280,"replication_time_us":5308037,"thread_start_us":24159,"threads_started":1}]]}
I20251024 19:48:13.837818 1616 tablet_service.cc:1505] Processing CreateTablet for tablet f37a01dc3d5f458d81605f2c41fa27dd (DEFAULT_TABLE table=default.Table_f16963cab00043a4887d93a6326a1d9c [id=bdfe9c0f34ac41eebb843bd17fa9635c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.847877 1766 tablet_bootstrap.cc:492] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:13.848388 1766 ts_tablet_manager.cc:1403] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.063s user 0.020s sys 0.000s
I20251024 19:48:13.852978 1613 tablet_service.cc:1505] Processing CreateTablet for tablet e0228d5b788e4c42bf4ecce876ecc3f4 (DEFAULT_TABLE table=default.Table_5afc602c5d154cf48565fa17d88d7c68 [id=cc27a6fdb94f4f6d876a9a627dcbe7be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.849509 1766 raft_consensus.cc:359] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.854382 1766 raft_consensus.cc:385] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:13.854562 1766 raft_consensus.cc:740] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:13.854935 1766 consensus_queue.cc:260] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.855875 1766 ts_tablet_manager.cc:1434] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251024 19:48:13.856581 1766 tablet_bootstrap.cc:492] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:13.858742 1766 tablet_bootstrap.cc:654] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:13.865445 1185 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 635a132735704cfda69bf7c8deaea81d (table default.Table_933a6697624149369ab3ec28462b466a [id=93a61862707b4a7ba8a0ff349704dc2b]). Expected version 1 got 0
I20251024 19:48:13.867334 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fd8aa1d9aca429dbfe46f472ebbf53b) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:13.868986 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1ea2bdfecf104bf18dacacebb8b5dcc1" } modify_external_catalogs: true
I20251024 19:48:13.869427 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1ea2bdfecf104bf18dacacebb8b5dcc1" } modify_external_catalogs: true
I20251024 19:48:13.873510 1766 tablet_bootstrap.cc:492] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:13.874154 1766 ts_tablet_manager.cc:1403] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20251024 19:48:13.875299 1766 raft_consensus.cc:359] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:13.876641 1766 raft_consensus.cc:385] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:13.876852 1766 raft_consensus.cc:740] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:13.878074 1766 consensus_queue.cc:260] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:13.878439 1447 consensus_peers.cc:597] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:13.879287 1766 ts_tablet_manager.cc:1434] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:48:13.881305 1766 tablet_bootstrap.cc:492] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:13.883097 1766 tablet_bootstrap.cc:654] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:13.885043 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f5ee199153824340833717cac0a3cea0" } modify_external_catalogs: true
I20251024 19:48:13.885490 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f5ee199153824340833717cac0a3cea0" } modify_external_catalogs: true
I20251024 19:48:13.529515 1774 rpcz_store.cc:276] 1024 19:48:11.376754 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.376808 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:13.169342 (+1792534us) ts_tablet_manager.cc:1151] Deleting tablet 2037fcca87b94b5fb31639b4d07307b0
1024 19:48:13.169349 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:13.524954 (+355605us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:13.528508 (+ 3554us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:13.529413 (+ 905us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1792322}
W20251024 19:48:13.893299 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 182314f0f3074150b139cb2476976e1a consensus state has no leader
I20251024 19:48:13.893729 1774 tablet_replica.cc:333] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:13.893751 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 2037fcca87b94b5fb31639b4d07307b0 (table default.Table_a4b63032d2224a6d95765cb93137f4a2 [id=d28fd80220c64f8496a09f5d4e712874]) successfully deleted
I20251024 19:48:13.893997 1774 raft_consensus.cc:2243] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:13.894188 1774 raft_consensus.cc:2272] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:13.895607 1774 ts_tablet_manager.cc:1916] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:13.896800 1766 tablet_bootstrap.cc:492] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:13.897558 1766 ts_tablet_manager.cc:1403] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20251024 19:48:13.898577 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fe00154d4ddf4d90a36ab30b672cc5cf" } modify_external_catalogs: true
I20251024 19:48:13.898921 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fe00154d4ddf4d90a36ab30b672cc5cf" } modify_external_catalogs: true
I20251024 19:48:13.899758 1766 raft_consensus.cc:359] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:13.900233 1766 raft_consensus.cc:385] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:13.900434 1766 raft_consensus.cc:740] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:13.900846 1766 consensus_queue.cc:260] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:13.901928 1766 ts_tablet_manager.cc:1434] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:13.903963 1766 tablet_bootstrap.cc:492] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:13.911586 1766 tablet_bootstrap.cc:654] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:13.913247 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet fce7de9523fb4878a58aa8aef0c6b182 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:13.916716 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ace18d554f14acc892bc956c30e9979 consensus state has no leader
I20251024 19:48:13.933449 1775 ts_tablet_manager.cc:1929] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251024 19:48:13.939216 1447 consensus_peers.cc:597] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:13.944085 1369 raft_consensus.cc:3060] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:13.953137 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1999b751e9344dcf81d26226c0589c8f consensus state has no leader
I20251024 19:48:13.963790 1335 tablet_service.cc:1505] Processing CreateTablet for tablet d7ea56637b2a429ba0ce95f35df42f20 (DEFAULT_TABLE table=default.Table_2ffcc8a68d3f46d381d6c565f890c6ad [id=8675c5f666914e81999fd8e471c3d3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.964210 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7ea56637b2a429ba0ce95f35df42f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.965107 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet b22b58404b8b441ba97ef536885a66e0 with delete_type TABLET_DATA_DELETED (replaced by tablet f37a01dc3d5f458d81605f2c41fa27dd) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:13.965647 1626 tablet_service.cc:1505] Processing CreateTablet for tablet 027e12c7961247a4a1664b85e4ddae4c (DEFAULT_TABLE table=default.Table_6b101c11690c47319e38598ef667655b [id=c071f24459334271be5e16b90bee2a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.966041 1696 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:13.969789 1617 tablet_service.cc:1505] Processing CreateTablet for tablet b74415cf7c7e47a696b454a5d44d8017 (DEFAULT_TABLE table=default.Table_f5ee199153824340833717cac0a3cea0 [id=41569aa93ac34c758747a9abc5483d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.970310 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce593bda7c8547bb9cf2b30ca6820776. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.973894 1488 tablet_service.cc:1505] Processing CreateTablet for tablet d7ea56637b2a429ba0ce95f35df42f20 (DEFAULT_TABLE table=default.Table_2ffcc8a68d3f46d381d6c565f890c6ad [id=8675c5f666914e81999fd8e471c3d3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.974323 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7ea56637b2a429ba0ce95f35df42f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.978546 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_80521123c9bb431dae3bf90e358d8276"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:13.978802 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_28cc3637c8e04cd291b968fe9631ae13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:13.979277 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28cc3637c8e04cd291b968fe9631ae13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.979593 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d56f14204c1b485fa6dffd55f3e7c95c" } modify_external_catalogs: true
I20251024 19:48:13.979612 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 913fee0ca74e4cf99d6697ad5c38f4eb (DEFAULT_TABLE table=default.Table_92ca5409f01446e494834187689f234d [id=212636f6527a4647a129d04c6126cdc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.979902 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d56f14204c1b485fa6dffd55f3e7c95c" } modify_external_catalogs: true
I20251024 19:48:13.980047 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913fee0ca74e4cf99d6697ad5c38f4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.981387 1344 tablet_service.cc:1505] Processing CreateTablet for tablet da3b9fc7313c42249c7f93e6728afcae (DEFAULT_TABLE table=default.Table_d27cfa82d3b04992b5e7d5d77c884000 [id=053088c5e2034a9e888ac24ea03aaef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.982514 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_af70ad0ef8864f3a83c8376e84e7ddd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:13.984056 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da3b9fc7313c42249c7f93e6728afcae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.984063 1483 tablet_service.cc:1505] Processing CreateTablet for tablet f405fc8974e243af982f30e00607d14a (DEFAULT_TABLE table=default.Table_ac33dd71eb504d6a8d92fef51601721e [id=b00dac3d67c3465d8be863986e0c69c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.984220 1477 tablet_service.cc:1505] Processing CreateTablet for tablet da3b9fc7313c42249c7f93e6728afcae (DEFAULT_TABLE table=default.Table_d27cfa82d3b04992b5e7d5d77c884000 [id=053088c5e2034a9e888ac24ea03aaef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.984433 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f405fc8974e243af982f30e00607d14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.984711 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da3b9fc7313c42249c7f93e6728afcae. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:13.979279 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80521123c9bb431dae3bf90e358d8276 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.989885 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 913fee0ca74e4cf99d6697ad5c38f4eb (DEFAULT_TABLE table=default.Table_92ca5409f01446e494834187689f234d [id=212636f6527a4647a129d04c6126cdc9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:13.990015 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af70ad0ef8864f3a83c8376e84e7ddd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:13.990278 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913fee0ca74e4cf99d6697ad5c38f4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:13.991935 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c737c6e46264439f8401a4a81d1fa4ae" } new_table_name: "default.Table_81cbf739a62f4e90a669a2638e7cc36f" modify_external_catalogs: true
I20251024 19:48:13.995119 1335 tablet_service.cc:1505] Processing CreateTablet for tablet f405fc8974e243af982f30e00607d14a (DEFAULT_TABLE table=default.Table_ac33dd71eb504d6a8d92fef51601721e [id=b00dac3d67c3465d8be863986e0c69c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:13.996039 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f405fc8974e243af982f30e00607d14a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:14.001152 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 71405.9us
I20251024 19:48:14.017750 1767 tablet_bootstrap.cc:492] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:14.028884 1766 tablet_bootstrap.cc:492] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:14.029172 1766 ts_tablet_manager.cc:1403] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.125s user 0.007s sys 0.011s
I20251024 19:48:14.029942 1766 raft_consensus.cc:359] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.030189 1766 raft_consensus.cc:385] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.030221 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 714) took 2317 ms. Trace:
I20251024 19:48:14.030274 1766 raft_consensus.cc:740] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.030505 1766 consensus_queue.cc:260] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.030448 1776 rpcz_store.cc:276] 1024 19:48:11.540610 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.540881 (+ 271us) service_pool.cc:225] Handling call
1024 19:48:13.085281 (+1544400us) ts_tablet_manager.cc:1151] Deleting tablet d9bf08ccf7054fedb9efcbb00354c71a
1024 19:48:13.085294 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:13.333533 (+248239us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:13.670047 (+336514us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:13.858012 (+187965us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27110,"tablet-delete.queue_time_us":1544167}
I20251024 19:48:14.032735 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet d9bf08ccf7054fedb9efcbb00354c71a (table default.Table_4a085504a0884b37b6787b3bcf7a847f [id=10833938cf88466b834698420d7c01e3]) successfully deleted
I20251024 19:48:14.035446 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 027e12c7961247a4a1664b85e4ddae4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.035974 1776 tablet_replica.cc:333] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:14.036160 1767 ts_tablet_manager.cc:1403] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.795s user 0.009s sys 0.017s
I20251024 19:48:14.036135 2824 consensus_queue.cc:1048] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.241s
I20251024 19:48:14.037137 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74415cf7c7e47a696b454a5d44d8017. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.057291 1767 raft_consensus.cc:359] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.057654 1767 raft_consensus.cc:385] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.057775 1767 raft_consensus.cc:740] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.058020 1767 consensus_queue.cc:260] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.058694 1767 ts_tablet_manager.cc:1434] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20251024 19:48:14.061468 1766 ts_tablet_manager.cc:1434] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.032s user 0.001s sys 0.000s
I20251024 19:48:14.064922 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 6e80dfddd21b41e98ebd226e0219b519 (DEFAULT_TABLE table=default.Table_5b54ca8cd83f49188876e061f70d629e [id=d587a8d80edb400daefac34b4175e4dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.065339 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e80dfddd21b41e98ebd226e0219b519. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.066689 1619 tablet_service.cc:1505] Processing CreateTablet for tablet f900b961f12e4e469ac3f440de0bf78e (DEFAULT_TABLE table=default.Table_8638421b2d894a4dbf4a9f06afc4c595 [id=5c65856731f2409abd549da6fa103672]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.067018 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f900b961f12e4e469ac3f440de0bf78e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.067051 1766 tablet_bootstrap.cc:492] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:14.067844 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 3c608ac55e9e42bc9c9b27beec717e30 (DEFAULT_TABLE table=default.Table_d61970c7cdd9475d9d6da8dc5f80a232 [id=bf39bf58617e4d36a1b1de926062b633]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.068152 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c608ac55e9e42bc9c9b27beec717e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.069042 1621 tablet_service.cc:1505] Processing CreateTablet for tablet e83527f7ff5c4e248f344a84515a5d6a (DEFAULT_TABLE table=default.Table_cde3670bf0d04dc0b2049193bcee10d4 [id=c3de6937e93a48dcb88cabee59a09d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.069658 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e83527f7ff5c4e248f344a84515a5d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.070432 1766 tablet_bootstrap.cc:654] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.071115 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 2346c03802a245a2af8c086b626d660d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.071348 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 73278db2097d4001a538d62dacb97c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.071568 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 9377725db312439e8a42dbd46da84a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.071729 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 2764689179504e079924fde7e425dbcd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.071892 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet d38154d3ffd34f4b8d7305b1c23a4fae with delete_type TABLET_DATA_DELETED (replaced by tablet 68ca2dd438d94bf0b5f1fc69f0a94517) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.072121 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 505e4fb0755a4caaa462e5f5154bfee2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b39e014607b4437b255c26c92c71861) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.072556 1614 tablet_service.cc:1505] Processing CreateTablet for tablet d7ea56637b2a429ba0ce95f35df42f20 (DEFAULT_TABLE table=default.Table_2ffcc8a68d3f46d381d6c565f890c6ad [id=8675c5f666914e81999fd8e471c3d3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.072986 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7ea56637b2a429ba0ce95f35df42f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.078162 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20e56d70049542f881f9490e4205dce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.080044 1767 tablet_bootstrap.cc:492] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:14.081921 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f37a01dc3d5f458d81605f2c41fa27dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.082216 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0228d5b788e4c42bf4ecce876ecc3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.083525 2838 raft_consensus.cc:493] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:14.083702 2838 raft_consensus.cc:515] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:14.084468 2838 leader_election.cc:290] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:14.089634 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:14.090098 1649 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.001s sys 0.018s
W20251024 19:48:14.090184 1649 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.713s user 0.001s sys 0.018s
W20251024 19:48:14.090282 1583 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.007s sys 0.010s
W20251024 19:48:14.090327 1583 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.614s user 0.007s sys 0.010s
I20251024 19:48:14.097167 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "758df87acbdb4adaa83e1ecbc44e7cf1" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:14.097461 1511 raft_consensus.cc:2393] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:14.099787 1775 log.cc:1199] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/5368789df6024d7daa9ccc2257137d58
I20251024 19:48:14.102947 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 913fee0ca74e4cf99d6697ad5c38f4eb (DEFAULT_TABLE table=default.Table_92ca5409f01446e494834187689f234d [id=212636f6527a4647a129d04c6126cdc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.103324 1775 ts_tablet_manager.cc:1950] T 5368789df6024d7daa9ccc2257137d58 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.103422 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913fee0ca74e4cf99d6697ad5c38f4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.104141 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 505) took 7373 ms. Trace:
I20251024 19:48:14.104564 1775 rpcz_store.cc:276] 1024 19:48:06.730435 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.736492 (+ 6057us) service_pool.cc:225] Handling call
1024 19:48:13.451114 (+6714622us) ts_tablet_manager.cc:1151] Deleting tablet 5368789df6024d7daa9ccc2257137d58
1024 19:48:13.451125 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:13.737427 (+286302us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:13.933322 (+195895us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.104115 (+170793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6714369}
I20251024 19:48:14.105955 1775 tablet_replica.cc:333] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.108405 1775 raft_consensus.cc:2243] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:14.109406 1622 tablet_service.cc:1505] Processing CreateTablet for tablet da3b9fc7313c42249c7f93e6728afcae (DEFAULT_TABLE table=default.Table_d27cfa82d3b04992b5e7d5d77c884000 [id=053088c5e2034a9e888ac24ea03aaef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.109853 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da3b9fc7313c42249c7f93e6728afcae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.110128 1775 raft_consensus.cc:2272] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.110782 1775 ts_tablet_manager.cc:1916] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.117777 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 5368789df6024d7daa9ccc2257137d58 (table default.Table_a81e7aae0a7a4f3db5c84efc4b16f873 [id=16bcec63f27d4dcd826ef11d45d58400]) successfully deleted
I20251024 19:48:14.125980 1775 ts_tablet_manager.cc:1929] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:14.126227 1775 log.cc:1199] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/9fffc42ddda14bd28800917ff2f0ced5
I20251024 19:48:14.126734 1775 ts_tablet_manager.cc:1950] T 9fffc42ddda14bd28800917ff2f0ced5 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.127616 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 518) took 7321 ms. Trace:
I20251024 19:48:14.128921 1775 rpcz_store.cc:276] 1024 19:48:06.806243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.806292 (+ 49us) service_pool.cc:225] Handling call
1024 19:48:14.105876 (+7299584us) ts_tablet_manager.cc:1151] Deleting tablet 9fffc42ddda14bd28800917ff2f0ced5
1024 19:48:14.105893 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.115718 (+ 9825us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.125244 (+ 9526us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.127593 (+ 2349us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7299321}
I20251024 19:48:14.129491 1775 tablet_replica.cc:333] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.129776 1775 raft_consensus.cc:2243] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:14.129974 1775 raft_consensus.cc:2272] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:14.130502 2640 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.003s sys 0.013s
W20251024 19:48:14.130602 2640 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.740s user 0.003s sys 0.013s
W20251024 19:48:14.130762 2747 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.000s sys 0.010s
I20251024 19:48:14.131191 1775 ts_tablet_manager.cc:1916] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.134169 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 9fffc42ddda14bd28800917ff2f0ced5 (table default.Table_64aaab6eda0a443fbb49223835af43e3 [id=52efb5f413d14db28314ecbb058a351f]) successfully deleted
I20251024 19:48:14.137095 1774 ts_tablet_manager.cc:1929] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:14.137213 1774 log.cc:1199] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/73ffe342f541447d8519ee24bc98a0f3
I20251024 19:48:14.137493 1774 ts_tablet_manager.cc:1950] T 73ffe342f541447d8519ee24bc98a0f3 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:14.138381 1184 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 635a132735704cfda69bf7c8deaea81d (table default.Table_933a6697624149369ab3ec28462b466a [id=93a61862707b4a7ba8a0ff349704dc2b]). Expected version 1 got 0
I20251024 19:48:14.138669 1184 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 187925082cdd4923a6e9148bf9f270fd (table default.Table_55f8c66bd97343438051a98af12d8c2c [id=33f5ac73298543d0b5af77a27e897b79]). Expected version 1 got 0
I20251024 19:48:14.138840 1184 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 4ffd446c269448d38cc8d987a53aecf9 (table default.Table_bedd8edc24014953ac8c7dcecf18f128 [id=2ae1c808837e48df97c4efb01a684da9]). Expected version 1 got 0
I20251024 19:48:14.139014 1184 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet e03c488d0f264684b0897a81a04d7466 (table default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8]). Expected version 2 got 0
I20251024 19:48:14.140208 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 07e75c7f35544bec82cb994dea5ee4ef
W20251024 19:48:14.142460 1305 leader_election.cc:343] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 758df87acbdb4adaa83e1ecbc44e7cf1
I20251024 19:48:14.142748 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1ff2aa8257594c2fa59a715522aea99d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:14.143357 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ff2aa8257594c2fa59a715522aea99d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:14.145190 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 07e75c7f35544bec82cb994dea5ee4ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.147331 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 07e75c7f35544bec82cb994dea5ee4ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.150082 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet bc31e6c25ad14947af20abd766f12e78 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:14.151067 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_abc327da99a44b9d9f01f12c54b1ede2" } new_table_name: "default.Table_5fe87e2cf20f469689cec015ffe87708" modify_external_catalogs: true
I20251024 19:48:14.151463 1775 ts_tablet_manager.cc:1929] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251024 19:48:14.130817 2747 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.687s user 0.000s sys 0.010s
I20251024 19:48:14.151861 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fffc42ddda14bd28800917ff2f0ced5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.152206 1444 leader_election.cc:304] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:14.159528 1775 log.cc:1199] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/ca0527af718f494a8bbcc66e7b325bed
I20251024 19:48:14.159942 1775 ts_tablet_manager.cc:1950] T ca0527af718f494a8bbcc66e7b325bed P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.160249 2615 raft_consensus.cc:2804] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:14.160423 2615 raft_consensus.cc:697] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:14.160773 2615 consensus_queue.cc:237] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.167845 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 519) took 7361 ms. Trace:
I20251024 19:48:14.168043 1775 rpcz_store.cc:276] 1024 19:48:06.806329 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.806574 (+ 245us) service_pool.cc:225] Handling call
1024 19:48:14.129443 (+7322869us) ts_tablet_manager.cc:1151] Deleting tablet ca0527af718f494a8bbcc66e7b325bed
1024 19:48:14.129452 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.147849 (+ 18397us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.151340 (+ 3491us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.167782 (+ 16442us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7322646}
I20251024 19:48:14.168802 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 520) took 7362 ms. Trace:
I20251024 19:48:14.168943 1775 rpcz_store.cc:276] 1024 19:48:06.806495 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.806828 (+ 333us) service_pool.cc:225] Handling call
1024 19:48:14.168639 (+7361811us) ts_tablet_manager.cc:1151] Deleting tablet f88390f2534e4799a6eaa56aa4245e97
1024 19:48:14.168654 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.168783 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7361619}
I20251024 19:48:14.169670 1775 tablet_replica.cc:333] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
W20251024 19:48:14.130849 1645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.000s sys 0.018s
I20251024 19:48:14.169929 1775 raft_consensus.cc:2243] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 2 LEADER]: Raft consensus shutting down.
W20251024 19:48:14.170014 1645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.689s user 0.000s sys 0.018s
I20251024 19:48:14.168654 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet ca0527af718f494a8bbcc66e7b325bed (table default.Table_51982d999b0847ec87fc62a473782a5c [id=9a05bcf7ad5d430c9ef6d6e18a27808e]) successfully deleted
I20251024 19:48:14.170454 1775 raft_consensus.cc:2272] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:14.170823 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet f88390f2534e4799a6eaa56aa4245e97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f88390f2534e4799a6eaa56aa4245e97 already in progress: creating tablet
I20251024 19:48:14.171221 1775 ts_tablet_manager.cc:1916] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.181365 1776 raft_consensus.cc:2243] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:14.181735 1776 raft_consensus.cc:2272] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.182854 1775 ts_tablet_manager.cc:1929] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251024 19:48:14.183051 1775 log.cc:1199] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/b6661b9e9a174849b6093d09cd8c1251
I20251024 19:48:14.183516 1775 ts_tablet_manager.cc:1950] T b6661b9e9a174849b6093d09cd8c1251 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.183943 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 517) took 7377 ms. Trace:
I20251024 19:48:14.184092 1775 rpcz_store.cc:276] 1024 19:48:06.806127 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:06.860074 (+ 53947us) service_pool.cc:225] Handling call
1024 19:48:14.169623 (+7309549us) ts_tablet_manager.cc:1151] Deleting tablet b6661b9e9a174849b6093d09cd8c1251
1024 19:48:14.169633 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.176542 (+ 6909us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.182741 (+ 6199us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.183919 (+ 1178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7309196}
I20251024 19:48:14.184983 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 527) took 7003 ms. Trace:
I20251024 19:48:14.185128 1775 rpcz_store.cc:276] 1024 19:48:07.181746 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.182332 (+ 586us) service_pool.cc:225] Handling call
1024 19:48:14.184839 (+7002507us) ts_tablet_manager.cc:1151] Deleting tablet 7deb97be82964ea0b2fe9caa867a6460
1024 19:48:14.184850 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.184963 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7002258}
I20251024 19:48:14.185740 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 529) took 7003 ms. Trace:
I20251024 19:48:14.185881 1775 rpcz_store.cc:276] 1024 19:48:07.182096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.182613 (+ 517us) service_pool.cc:225] Handling call
1024 19:48:14.185651 (+7003038us) ts_tablet_manager.cc:1151] Deleting tablet e20881e400cf4fac914b70afc79c084f
1024 19:48:14.185662 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.185723 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7002845}
I20251024 19:48:14.186416 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 528) took 7004 ms. Trace:
I20251024 19:48:14.186566 1775 rpcz_store.cc:276] 1024 19:48:07.181934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.183933 (+ 1999us) service_pool.cc:225] Handling call
1024 19:48:14.186306 (+7002373us) ts_tablet_manager.cc:1151] Deleting tablet fc7ccf1495924782b9b6ac1173bc73a1
1024 19:48:14.186318 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.186399 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7002145}
I20251024 19:48:14.187280 1775 tablet_replica.cc:333] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.187594 1775 raft_consensus.cc:2243] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:14.188040 1775 pending_rounds.cc:70] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e: Trying to abort 1 pending ops.
I20251024 19:48:14.188153 1775 pending_rounds.cc:77] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429340838838272 op_type: NO_OP noop_request { }
I20251024 19:48:14.188182 1623 tablet_service.cc:1505] Processing CreateTablet for tablet f405fc8974e243af982f30e00607d14a (DEFAULT_TABLE table=default.Table_ac33dd71eb504d6a8d92fef51601721e [id=b00dac3d67c3465d8be863986e0c69c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.188326 1775 raft_consensus.cc:2889] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:14.188530 1775 raft_consensus.cc:2272] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.188772 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f405fc8974e243af982f30e00607d14a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:14.188915 1646 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.002s sys 0.015s
W20251024 19:48:14.189002 1646 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.731s user 0.002s sys 0.015s
I20251024 19:48:14.189514 1775 ts_tablet_manager.cc:1916] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.193125 2857 raft_consensus.cc:493] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:14.193356 2857 raft_consensus.cc:515] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:14.194325 2857 leader_election.cc:290] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:14.199039 1766 tablet_bootstrap.cc:492] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:14.199268 1766 ts_tablet_manager.cc:1403] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.132s user 0.004s sys 0.033s
I20251024 19:48:14.200408 1766 raft_consensus.cc:359] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.200599 1766 raft_consensus.cc:385] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.200670 1766 raft_consensus.cc:740] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.200876 1766 consensus_queue.cc:260] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.202111 1766 ts_tablet_manager.cc:1434] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251024 19:48:14.202459 1766 tablet_bootstrap.cc:492] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:14.204550 1766 tablet_bootstrap.cc:654] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.206662 1776 ts_tablet_manager.cc:1916] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.209688 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet b6661b9e9a174849b6093d09cd8c1251 (table default.Table_7dbe5740202642f1be406c4897a72541 [id=df94703d2472430387b81c028130d628]) successfully deleted
W20251024 19:48:14.210587 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7deb97be82964ea0b2fe9caa867a6460 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7deb97be82964ea0b2fe9caa867a6460 already in progress: creating tablet
W20251024 19:48:14.210911 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet e20881e400cf4fac914b70afc79c084f because the tablet was not found. No further retry: Not found: Tablet not found: e20881e400cf4fac914b70afc79c084f
W20251024 19:48:14.211130 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet fc7ccf1495924782b9b6ac1173bc73a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc7ccf1495924782b9b6ac1173bc73a1 already in progress: creating tablet
I20251024 19:48:14.211431 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ffd446c269448d38cc8d987a53aecf9" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:14.211639 1649 raft_consensus.cc:2468] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:14.220057 2824 raft_consensus.cc:493] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:14.220237 2824 raft_consensus.cc:515] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:14.220801 2824 leader_election.cc:290] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:14.225129 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce8c179e98944df0b510a0e8daf3920f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:14.225481 1646 raft_consensus.cc:2468] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:14.227221 1775 ts_tablet_manager.cc:1929] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:14.228267 1766 tablet_bootstrap.cc:492] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:14.229437 1775 log.cc:1199] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/5b96fc2309274048bed7a35e4bf4bae2
I20251024 19:48:14.229468 1766 ts_tablet_manager.cc:1403] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20251024 19:48:14.230293 1775 ts_tablet_manager.cc:1950] T 5b96fc2309274048bed7a35e4bf4bae2 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.230834 1766 raft_consensus.cc:359] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.233023 1766 raft_consensus.cc:385] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.233114 1766 raft_consensus.cc:740] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.233413 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 530) took 6920 ms. Trace:
I20251024 19:48:14.233345 1766 consensus_queue.cc:260] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.233541 1775 rpcz_store.cc:276] 1024 19:48:07.312638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.313084 (+ 446us) service_pool.cc:225] Handling call
1024 19:48:14.187228 (+6874144us) ts_tablet_manager.cc:1151] Deleting tablet 5b96fc2309274048bed7a35e4bf4bae2
1024 19:48:14.187238 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.201375 (+ 14137us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.227110 (+ 25735us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.233397 (+ 6287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6871044}
I20251024 19:48:14.233976 1775 tablet_replica.cc:333] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.234026 1766 ts_tablet_manager.cc:1434] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:14.234201 1775 raft_consensus.cc:2243] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:14.234294 1766 tablet_bootstrap.cc:492] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:14.234606 1775 pending_rounds.cc:70] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e: Trying to abort 1 pending ops.
I20251024 19:48:14.234688 1775 pending_rounds.cc:77] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429336832106496 op_type: NO_OP noop_request { }
I20251024 19:48:14.234797 1775 raft_consensus.cc:2889] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:14.234905 1775 raft_consensus.cc:2272] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.235488 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 5b96fc2309274048bed7a35e4bf4bae2 (table default.Table_dc366a1f70d84d4f8276c1872b9fa0f7 [id=0f190205e9b347f6906d6f86b0624681]) successfully deleted
I20251024 19:48:14.235669 1775 ts_tablet_manager.cc:1916] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.239619 1776 ts_tablet_manager.cc:1929] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:14.242897 1775 ts_tablet_manager.cc:1929] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:14.245690 1766 tablet_bootstrap.cc:654] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.256884 1775 log.cc:1199] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/d76410afd7ba4c73836f4791ef739056
I20251024 19:48:14.258118 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d38154d3ffd34f4b8d7305b1c23a4fae" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:14.258325 1511 raft_consensus.cc:2468] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:14.258515 1776 log.cc:1199] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/9b89a90bb4ca447fbaf34fe39001c261
I20251024 19:48:14.257432 1775 ts_tablet_manager.cc:1950] T d76410afd7ba4c73836f4791ef739056 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.259431 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 531) took 6943 ms. Trace:
I20251024 19:48:14.263062 1775 rpcz_store.cc:276] 1024 19:48:07.316364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.316425 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:14.233930 (+6917505us) ts_tablet_manager.cc:1151] Deleting tablet d76410afd7ba4c73836f4791ef739056
1024 19:48:14.233940 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.239494 (+ 5554us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.242800 (+ 3306us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.259409 (+ 16609us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6917302}
I20251024 19:48:14.263664 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet d76410afd7ba4c73836f4791ef739056 (table default.Table_34bbddd692a44f46a4925c957f4e8848 [id=6f9b0ed7552c4556afd0f59316831a91]) successfully deleted
I20251024 19:48:14.265471 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 532) took 6907 ms. Trace:
I20251024 19:48:14.265674 1775 rpcz_store.cc:276] 1024 19:48:07.358290 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.358346 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:14.265321 (+6906975us) ts_tablet_manager.cc:1151] Deleting tablet 0266cef4e2034303978d7456c47ea98c
1024 19:48:14.265333 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.265447 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6906750}
I20251024 19:48:14.266583 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 533) took 6899 ms. Trace:
I20251024 19:48:14.266762 1775 rpcz_store.cc:276] 1024 19:48:07.366994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.367050 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:14.266461 (+6899411us) ts_tablet_manager.cc:1151] Deleting tablet 4193750331874d7ba9bbe6279c3a8e37
1024 19:48:14.266471 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.266563 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6899221}
W20251024 19:48:14.267314 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4193750331874d7ba9bbe6279c3a8e37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4193750331874d7ba9bbe6279c3a8e37 already in progress: creating tablet
I20251024 19:48:14.267765 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 534) took 6852 ms. Trace:
I20251024 19:48:14.267902 1775 rpcz_store.cc:276] 1024 19:48:07.414992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.415047 (+ 55us) service_pool.cc:225] Handling call
1024 19:48:14.267642 (+6852595us) ts_tablet_manager.cc:1151] Deleting tablet 7712392f200842c4b728e019d6a3cfb0
1024 19:48:14.267661 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.267749 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6851546}
W20251024 19:48:14.268533 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7712392f200842c4b728e019d6a3cfb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7712392f200842c4b728e019d6a3cfb0 already in progress: creating tablet
I20251024 19:48:14.268875 1775 tablet_replica.cc:333] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.269111 1775 raft_consensus.cc:2243] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:14.269326 1775 raft_consensus.cc:2272] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.269948 1775 ts_tablet_manager.cc:1916] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.272301 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b74415cf7c7e47a696b454a5d44d8017
I20251024 19:48:14.272511 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b19cce078dc7426aacbb60f45379e506
I20251024 19:48:14.274143 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet b74415cf7c7e47a696b454a5d44d8017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.275527 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet b19cce078dc7426aacbb60f45379e506 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.275686 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e5d0d22a76d5435c8861460346530bd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:14.276376 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5d0d22a76d5435c8861460346530bd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:14.279265 1775 ts_tablet_manager.cc:1929] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:14.280830 1775 log.cc:1199] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/ce8c179e98944df0b510a0e8daf3920f
I20251024 19:48:14.281318 1775 ts_tablet_manager.cc:1950] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.282014 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 536) took 6844 ms. Trace:
I20251024 19:48:14.282150 1775 rpcz_store.cc:276] 1024 19:48:07.437409 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.437481 (+ 72us) service_pool.cc:225] Handling call
1024 19:48:14.268828 (+6831347us) ts_tablet_manager.cc:1151] Deleting tablet ce8c179e98944df0b510a0e8daf3920f
1024 19:48:14.268837 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.275247 (+ 6410us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.279147 (+ 3900us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.281993 (+ 2846us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6830978}
I20251024 19:48:14.282752 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet ce8c179e98944df0b510a0e8daf3920f (table default.Table_dedd7e29ac324761b962fbd47c7c25e9 [id=870c629198ed4f53b9317b1340e75ea2]) successfully deleted
I20251024 19:48:14.283226 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 535) took 6846 ms. Trace:
I20251024 19:48:14.283360 1775 rpcz_store.cc:276] 1024 19:48:07.437219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.437502 (+ 283us) service_pool.cc:225] Handling call
1024 19:48:14.283094 (+6845592us) ts_tablet_manager.cc:1151] Deleting tablet d4b3a50db1714eb38cc0ea03f612fc95
1024 19:48:14.283105 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.283207 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6845084}
W20251024 19:48:14.283875 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet d4b3a50db1714eb38cc0ea03f612fc95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4b3a50db1714eb38cc0ea03f612fc95 already in progress: creating tablet
I20251024 19:48:14.284228 1775 tablet_replica.cc:333] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.284482 1775 raft_consensus.cc:2243] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:14.284852 1775 raft_consensus.cc:2272] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.285482 1775 ts_tablet_manager.cc:1916] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.288297 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet d76410afd7ba4c73836f4791ef739056 with delete_type TABLET_DATA_DELETED (replaced by tablet b153f75caebb4d869e77529c9c9bb50d) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.288524 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet d38154d3ffd34f4b8d7305b1c23a4fae with delete_type TABLET_DATA_DELETED (replaced by tablet 68ca2dd438d94bf0b5f1fc69f0a94517) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.289394 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet b6661b9e9a174849b6093d09cd8c1251 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.289415 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f with delete_type TABLET_DATA_DELETED (replaced by tablet 9a1031d785b646b095ece9be666d2a77) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.289819 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 505e4fb0755a4caaa462e5f5154bfee2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b39e014607b4437b255c26c92c71861) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.290928 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet cf18f375266b4448b3f67afd787df6ad
I20251024 19:48:14.295706 1775 ts_tablet_manager.cc:1929] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:14.296622 1775 log.cc:1199] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/5fd7726c532145c6b44ea9a37544510d
I20251024 19:48:14.296875 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet cf18f375266b4448b3f67afd787df6ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.297165 1775 ts_tablet_manager.cc:1950] T 5fd7726c532145c6b44ea9a37544510d P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.298091 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 537) took 6850 ms. Trace:
I20251024 19:48:14.299283 1775 rpcz_store.cc:276] 1024 19:48:07.447745 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.447813 (+ 68us) service_pool.cc:225] Handling call
1024 19:48:14.284178 (+6836365us) ts_tablet_manager.cc:1151] Deleting tablet 5fd7726c532145c6b44ea9a37544510d
1024 19:48:14.284187 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.291946 (+ 7759us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.295587 (+ 3641us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.298071 (+ 2484us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6836072}
I20251024 19:48:14.299762 1775 tablet_replica.cc:333] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.300009 1775 raft_consensus.cc:2243] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:14.300238 1775 raft_consensus.cc:2272] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.300841 1775 ts_tablet_manager.cc:1916] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.303485 1189 master_service.cc:954] ReplaceTablet: received request to replace tablet b3e9960b7821421aa23e7ef5389f894a from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:14.309309 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 839cef50795840fab564b94d88565f8c
I20251024 19:48:14.310662 1775 ts_tablet_manager.cc:1929] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:14.310884 1775 log.cc:1199] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/73ffe342f541447d8519ee24bc98a0f3
I20251024 19:48:14.311092 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 839cef50795840fab564b94d88565f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.311074 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet cf18f375266b4448b3f67afd787df6ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.311360 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 5fd7726c532145c6b44ea9a37544510d (table default.Table_999546c6f492460bb6dc96ab55b7d3ad [id=7dbbe53e70a547888a510250e0583e1d]) successfully deleted
I20251024 19:48:14.311321 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet b19cce078dc7426aacbb60f45379e506 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.311908 1775 ts_tablet_manager.cc:1950] T 73ffe342f541447d8519ee24bc98a0f3 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.312378 1305 leader_election.cc:304] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:14.312573 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 839cef50795840fab564b94d88565f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.312975 2838 raft_consensus.cc:3060] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:14.313073 1305 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 170623us
I20251024 19:48:14.313485 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d38154d3ffd34f4b8d7305b1c23a4fae" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:14.315872 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 540) took 6808 ms. Trace:
I20251024 19:48:14.316046 1775 rpcz_store.cc:276] 1024 19:48:07.507661 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.507716 (+ 55us) service_pool.cc:225] Handling call
1024 19:48:14.299706 (+6791990us) ts_tablet_manager.cc:1151] Deleting tablet 73ffe342f541447d8519ee24bc98a0f3
1024 19:48:14.299716 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.304535 (+ 4819us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.310532 (+ 5997us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.315846 (+ 5314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6791793}
I20251024 19:48:14.317193 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 73ffe342f541447d8519ee24bc98a0f3 (table default.Table_74e4f24f12db403686548924c83c5191 [id=508d6cdc5d4a494bad31eaf36383e214]) successfully deleted
I20251024 19:48:14.317692 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 539) took 6818 ms. Trace:
I20251024 19:48:14.317831 1775 rpcz_store.cc:276] 1024 19:48:07.499035 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.510937 (+ 11902us) service_pool.cc:225] Handling call
1024 19:48:14.317530 (+6806593us) ts_tablet_manager.cc:1151] Deleting tablet 4cac515c0a00403fb680454c1721be0a
1024 19:48:14.317541 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.317676 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6805735}
W20251024 19:48:14.318457 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4cac515c0a00403fb680454c1721be0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cac515c0a00403fb680454c1721be0a already in progress: creating tablet
I20251024 19:48:14.318970 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 545) took 6734 ms. Trace:
W20251024 19:48:14.311455 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 45354.3us
I20251024 19:48:14.319432 1586 leader_election.cc:304] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:14.320542 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet b19cce078dc7426aacbb60f45379e506 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:14.315225 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d38154d3ffd34f4b8d7305b1c23a4fae" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:14.321055 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 07e75c7f35544bec82cb994dea5ee4ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:14.321228 1333 tablet_service.cc:1552] Processing DeleteTablet for tablet b74415cf7c7e47a696b454a5d44d8017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:14.321645 1768 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.050s sys 0.073s
W20251024 19:48:14.321750 1768 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.538s user 0.050s sys 0.073s
I20251024 19:48:14.321792 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet cf18f375266b4448b3f67afd787df6ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:14.322640 1333 tablet_service.cc:1552] Processing DeleteTablet for tablet 839cef50795840fab564b94d88565f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:14.331647 1768 tablet_bootstrap.cc:492] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:14.331844 1768 ts_tablet_manager.cc:1403] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.550s user 0.050s sys 0.073s
I20251024 19:48:14.332953 1768 raft_consensus.cc:359] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.333185 1768 raft_consensus.cc:385] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.333251 1768 raft_consensus.cc:740] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.333559 1768 consensus_queue.cc:260] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:14.322955 1447 leader_election.cc:343] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:14.344930 1775 rpcz_store.cc:276] 1024 19:48:07.584460 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:07.585031 (+ 571us) service_pool.cc:225] Handling call
1024 19:48:14.318865 (+6733834us) ts_tablet_manager.cc:1151] Deleting tablet 1295c99836d14a1cb0002e6c6aaee1eb
1024 19:48:14.318875 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.318956 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6733613}
W20251024 19:48:14.346079 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 1295c99836d14a1cb0002e6c6aaee1eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1295c99836d14a1cb0002e6c6aaee1eb already in progress: creating tablet
I20251024 19:48:14.347308 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 568) took 6319 ms. Trace:
I20251024 19:48:14.347856 2828 raft_consensus.cc:2804] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:14.347949 2828 raft_consensus.cc:493] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:14.348008 2828 raft_consensus.cc:3060] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.348337 1768 ts_tablet_manager.cc:1434] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.016s user 0.001s sys 0.008s
I20251024 19:48:14.348651 1768 tablet_bootstrap.cc:492] T 03cdbd1b62b741249ad78562a7edfd22 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:14.350293 1775 rpcz_store.cc:276] 1024 19:48:08.027678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.027961 (+ 283us) service_pool.cc:225] Handling call
1024 19:48:14.346892 (+6318931us) ts_tablet_manager.cc:1151] Deleting tablet bc61fded84b24e00b299e5d342076708
1024 19:48:14.347143 (+ 251us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.347286 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6310998}
I20251024 19:48:14.350528 1768 tablet_bootstrap.cc:654] T 03cdbd1b62b741249ad78562a7edfd22 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.350591 1369 raft_consensus.cc:3060] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:14.351504 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet bc61fded84b24e00b299e5d342076708 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc61fded84b24e00b299e5d342076708 already in progress: creating tablet
I20251024 19:48:14.352464 2808 raft_consensus.cc:493] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:14.352602 2808 raft_consensus.cc:515] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.353212 2808 leader_election.cc:290] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:14.354132 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c77f6bbf7b145389ca6cf74ddd3aa3f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:14.354354 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 569) took 6316 ms. Trace:
W20251024 19:48:14.354794 1447 leader_election.cc:343] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:14.356436 1586 leader_election.cc:343] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:14.354437 1775 rpcz_store.cc:276] 1024 19:48:08.037651 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.038151 (+ 500us) service_pool.cc:225] Handling call
1024 19:48:14.351938 (+6313787us) ts_tablet_manager.cc:1151] Deleting tablet 8af55c71122f4712ba1050e42f01ed83
1024 19:48:14.351950 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.354333 (+ 2383us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6308439}
W20251024 19:48:14.358644 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 8af55c71122f4712ba1050e42f01ed83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8af55c71122f4712ba1050e42f01ed83 already in progress: creating tablet
I20251024 19:48:14.361869 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 570) took 6305 ms. Trace:
I20251024 19:48:14.362422 1775 rpcz_store.cc:276] 1024 19:48:08.056764 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.056883 (+ 119us) service_pool.cc:225] Handling call
1024 19:48:14.357702 (+6300819us) ts_tablet_manager.cc:1151] Deleting tablet c4d261f5f55949159aa1b72797e20fd5
1024 19:48:14.357713 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.361844 (+ 4131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6271553}
W20251024 19:48:14.363448 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet c4d261f5f55949159aa1b72797e20fd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4d261f5f55949159aa1b72797e20fd5 already in progress: creating tablet
I20251024 19:48:14.364028 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 572) took 6235 ms. Trace:
I20251024 19:48:14.364419 1768 tablet_bootstrap.cc:492] T 03cdbd1b62b741249ad78562a7edfd22 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:14.364581 1768 ts_tablet_manager.cc:1403] T 03cdbd1b62b741249ad78562a7edfd22 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20251024 19:48:14.364598 1775 rpcz_store.cc:276] 1024 19:48:08.128261 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.129425 (+ 1164us) service_pool.cc:225] Handling call
1024 19:48:14.363907 (+6234482us) ts_tablet_manager.cc:1151] Deleting tablet 55986287a8bb40c8bb28e536fb22fb27
1024 19:48:14.363919 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.364012 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6212113}
I20251024 19:48:14.365504 1768 raft_consensus.cc:359] T 03cdbd1b62b741249ad78562a7edfd22 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.365691 1768 raft_consensus.cc:385] T 03cdbd1b62b741249ad78562a7edfd22 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.365746 1768 raft_consensus.cc:740] T 03cdbd1b62b741249ad78562a7edfd22 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.365918 1768 consensus_queue.cc:260] T 03cdbd1b62b741249ad78562a7edfd22 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.366102 1776 ts_tablet_manager.cc:1950] T 9b89a90bb4ca447fbaf34fe39001c261 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:14.366232 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 573) took 6237 ms. Trace:
I20251024 19:48:14.366601 1775 rpcz_store.cc:276] 1024 19:48:08.128383 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.129428 (+ 1045us) service_pool.cc:225] Handling call
1024 19:48:14.365966 (+6236538us) ts_tablet_manager.cc:1151] Deleting tablet c1cdd5ec32ca49c1bfafaf3a24050e89
1024 19:48:14.365975 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.366216 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6173261}
I20251024 19:48:14.366861 1768 ts_tablet_manager.cc:1434] T 03cdbd1b62b741249ad78562a7edfd22 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251024 19:48:14.367103 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1cdd5ec32ca49c1bfafaf3a24050e89 already in progress: creating tablet
I20251024 19:48:14.367203 1768 tablet_bootstrap.cc:492] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:14.367410 1775 tablet_replica.cc:333] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.367635 1775 raft_consensus.cc:2243] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:14.367854 1775 raft_consensus.cc:2272] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.368507 1775 ts_tablet_manager.cc:1916] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.369195 1768 tablet_bootstrap.cc:654] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:14.372465 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec11453f49054f9dbbe620374e59a6ce consensus state has no leader
I20251024 19:48:14.374107 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_738f1cac13cb4cff9e3d7e3a36be4af1" } new_table_name: "default.Table_d5f808e243064ca48895273f0987340a" modify_external_catalogs: true
I20251024 19:48:14.375882 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d25c3d97a9a54883942fe92efe382f58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:14.376591 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d25c3d97a9a54883942fe92efe382f58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:14.376991 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8840cc4de84c4e1c95015f5440515ed7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:14.377610 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8840cc4de84c4e1c95015f5440515ed7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:14.377662 1775 ts_tablet_manager.cc:1929] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:14.377892 1775 log.cc:1199] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/2037fcca87b94b5fb31639b4d07307b0
I20251024 19:48:14.378290 1775 ts_tablet_manager.cc:1950] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.378844 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 571) took 6291 ms. Trace:
I20251024 19:48:14.378979 1775 rpcz_store.cc:276] 1024 19:48:08.087769 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.091662 (+ 3893us) service_pool.cc:225] Handling call
1024 19:48:14.367367 (+6275705us) ts_tablet_manager.cc:1151] Deleting tablet 2037fcca87b94b5fb31639b4d07307b0
1024 19:48:14.367375 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.373131 (+ 5756us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.377517 (+ 4386us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.378823 (+ 1306us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6172075}
I20251024 19:48:14.379595 1768 tablet_bootstrap.cc:492] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:14.379774 1768 ts_tablet_manager.cc:1403] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20251024 19:48:14.379765 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 2037fcca87b94b5fb31639b4d07307b0 (table default.Table_a4b63032d2224a6d95765cb93137f4a2 [id=d28fd80220c64f8496a09f5d4e712874]) successfully deleted
I20251024 19:48:14.380244 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 574) took 6151 ms. Trace:
I20251024 19:48:14.380365 1775 rpcz_store.cc:276] 1024 19:48:08.228584 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.228795 (+ 211us) service_pool.cc:225] Handling call
1024 19:48:14.380091 (+6151296us) ts_tablet_manager.cc:1151] Deleting tablet 57f522a825264ea7b215304dbb7aa2d5
1024 19:48:14.380105 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.380224 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6142769}
I20251024 19:48:14.380595 1768 raft_consensus.cc:359] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.380618 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1f888bb5a760477c88e82d58b11aa364"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:14.380779 1768 raft_consensus.cc:385] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.380828 1768 raft_consensus.cc:740] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.380949 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2622ff0be0624668b5534e85df77e487" } new_table_name: "default.Table_3dbd05fe93094e11baed3b6447a56980" modify_external_catalogs: true
I20251024 19:48:14.381024 1768 consensus_queue.cc:260] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:14.381135 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f888bb5a760477c88e82d58b11aa364 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:14.380923 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 57f522a825264ea7b215304dbb7aa2d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57f522a825264ea7b215304dbb7aa2d5 already in progress: creating tablet
I20251024 19:48:14.381610 1775 tablet_replica.cc:333] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.381841 1775 raft_consensus.cc:2243] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:14.382047 1775 raft_consensus.cc:2272] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.382732 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9ee642f4c0344db6b73e09e30ab6d351" } new_table_name: "default.Table_d3ef2e51cc8046d9baf9f18c54da359c" modify_external_catalogs: true
I20251024 19:48:14.383669 1768 ts_tablet_manager.cc:1434] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251024 19:48:14.383970 1768 tablet_bootstrap.cc:492] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:14.383975 1775 ts_tablet_manager.cc:1916] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.384368 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b6ce782f9e28425585cf22bc3c1bc98b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:14.385010 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6ce782f9e28425585cf22bc3c1bc98b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:14.385095 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fe7a480771e74942ab56f7fdd69ecf23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:14.385565 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe7a480771e74942ab56f7fdd69ecf23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:14.385736 1768 tablet_bootstrap.cc:654] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.391001 1768 tablet_bootstrap.cc:492] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:14.391132 1768 ts_tablet_manager.cc:1403] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20251024 19:48:14.391381 1775 ts_tablet_manager.cc:1929] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:14.391681 1775 log.cc:1199] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/fca73ab61f0c47df9186459caf2b9a32
I20251024 19:48:14.392061 1775 ts_tablet_manager.cc:1950] T fca73ab61f0c47df9186459caf2b9a32 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.391993 1768 raft_consensus.cc:359] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.392212 1768 raft_consensus.cc:385] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.392282 1768 raft_consensus.cc:740] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.392513 1768 consensus_queue.cc:260] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.393344 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 575) took 6031 ms. Trace:
I20251024 19:48:14.393471 1775 rpcz_store.cc:276] 1024 19:48:08.361773 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.366363 (+ 4590us) service_pool.cc:225] Handling call
1024 19:48:14.381538 (+6015175us) ts_tablet_manager.cc:1151] Deleting tablet fca73ab61f0c47df9186459caf2b9a32
1024 19:48:14.381548 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.388318 (+ 6770us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.391250 (+ 2932us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.393323 (+ 2073us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6014517}
I20251024 19:48:14.394047 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet fca73ab61f0c47df9186459caf2b9a32 (table default.Table_420ebc050cf440e9a59a18e26f7b83af [id=be1cc9d047554ea6a9afa531fb08a81d]) successfully deleted
I20251024 19:48:14.394531 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 596) took 5487 ms. Trace:
I20251024 19:48:14.394652 1775 rpcz_store.cc:276] 1024 19:48:08.907309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.907369 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:14.394407 (+5487038us) ts_tablet_manager.cc:1151] Deleting tablet 4c54e569156042fca5e9ab0feaedf632
1024 19:48:14.394416 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.394514 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5475704}
W20251024 19:48:14.395133 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4c54e569156042fca5e9ab0feaedf632 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c54e569156042fca5e9ab0feaedf632 already in progress: creating tablet
I20251024 19:48:14.395534 1768 ts_tablet_manager.cc:1434] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20251024 19:48:14.395635 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 597) took 5454 ms. Trace:
I20251024 19:48:14.395831 1768 tablet_bootstrap.cc:492] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:14.395781 1775 rpcz_store.cc:276] 1024 19:48:08.941640 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.941720 (+ 80us) service_pool.cc:225] Handling call
1024 19:48:14.395485 (+5453765us) ts_tablet_manager.cc:1151] Deleting tablet b240965605904533bd2af99f61aaef26
1024 19:48:14.395493 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.395574 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5430703}
W20251024 19:48:14.396302 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b240965605904533bd2af99f61aaef26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b240965605904533bd2af99f61aaef26 already in progress: creating tablet
I20251024 19:48:14.396790 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 599) took 5409 ms. Trace:
I20251024 19:48:14.396904 1775 rpcz_store.cc:276] 1024 19:48:08.987855 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.987930 (+ 75us) service_pool.cc:225] Handling call
1024 19:48:14.396674 (+5408744us) ts_tablet_manager.cc:1151] Deleting tablet bbbc3f1886e744ab9450a0f5776721d9
1024 19:48:14.396683 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.396773 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5397631}
W20251024 19:48:14.397475 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet bbbc3f1886e744ab9450a0f5776721d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbbc3f1886e744ab9450a0f5776721d9 already in progress: creating tablet
I20251024 19:48:14.397826 1775 tablet_replica.cc:333] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.398116 1775 raft_consensus.cc:2243] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:14.398319 1775 raft_consensus.cc:2272] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.398918 1775 ts_tablet_manager.cc:1916] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.399703 1768 tablet_bootstrap.cc:654] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.400152 1619 tablet_service.cc:1552] Processing DeleteTablet for tablet b74415cf7c7e47a696b454a5d44d8017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.404641 2864 raft_consensus.cc:493] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:14.404783 2864 raft_consensus.cc:515] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.406052 2864 leader_election.cc:290] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:14.405714 1775 ts_tablet_manager.cc:1929] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:14.406121 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b88a6a5237d146f5b6f082ca3af25b2b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:14.406915 1775 log.cc:1199] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/f51c56337ebb450aaf62dcc81ef63ae8
I20251024 19:48:14.407301 1775 ts_tablet_manager.cc:1950] T f51c56337ebb450aaf62dcc81ef63ae8 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.408507 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 598) took 5430 ms. Trace:
I20251024 19:48:14.408643 1775 rpcz_store.cc:276] 1024 19:48:08.977766 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:08.985060 (+ 7294us) service_pool.cc:225] Handling call
1024 19:48:14.397783 (+5412723us) ts_tablet_manager.cc:1151] Deleting tablet f51c56337ebb450aaf62dcc81ef63ae8
1024 19:48:14.397792 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.402200 (+ 4408us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.405614 (+ 3414us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.408486 (+ 2872us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5386783}
I20251024 19:48:14.409190 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet f51c56337ebb450aaf62dcc81ef63ae8 (table default.Table_efcd2af1913c4a73ba5148272aa880aa [id=01a8150e819b4075810a95720eaa5840]) successfully deleted
I20251024 19:48:14.409641 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 600) took 5402 ms. Trace:
I20251024 19:48:14.409773 1775 rpcz_store.cc:276] 1024 19:48:09.007212 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.007992 (+ 780us) service_pool.cc:225] Handling call
1024 19:48:14.409489 (+5401497us) ts_tablet_manager.cc:1151] Deleting tablet 98dd8bacd8374f469799872859db4206
1024 19:48:14.409499 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.409623 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5387630}
W20251024 19:48:14.410267 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 98dd8bacd8374f469799872859db4206 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98dd8bacd8374f469799872859db4206 already in progress: creating tablet
I20251024 19:48:14.410658 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 602) took 5400 ms. Trace:
I20251024 19:48:14.411144 1775 rpcz_store.cc:276] 1024 19:48:09.011023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.046467 (+ 35444us) service_pool.cc:225] Handling call
1024 19:48:14.410554 (+5364087us) ts_tablet_manager.cc:1151] Deleting tablet 23d4dd0b42d343cb9354ac698a8c761f
1024 19:48:14.410563 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.410643 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5295497}
I20251024 19:48:14.411851 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 605) took 5335 ms. Trace:
I20251024 19:48:14.411980 1775 rpcz_store.cc:276] 1024 19:48:09.076558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.076626 (+ 68us) service_pool.cc:225] Handling call
1024 19:48:14.411732 (+5335106us) ts_tablet_manager.cc:1151] Deleting tablet 2bb5f84c1f104b338be71cb55ae77929
1024 19:48:14.411741 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.411834 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5295219}
I20251024 19:48:14.412216 1768 tablet_bootstrap.cc:492] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:14.412376 1768 ts_tablet_manager.cc:1403] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
W20251024 19:48:14.412498 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2bb5f84c1f104b338be71cb55ae77929 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bb5f84c1f104b338be71cb55ae77929 already in progress: creating tablet
I20251024 19:48:14.412922 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 606) took 5254 ms. Trace:
I20251024 19:48:14.413038 1775 rpcz_store.cc:276] 1024 19:48:09.158052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.158110 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:14.412805 (+5254695us) ts_tablet_manager.cc:1151] Deleting tablet e68997931ede4b6398570a7d08edaf4d
1024 19:48:14.412816 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.412904 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5239919}
W20251024 19:48:14.413470 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet e68997931ede4b6398570a7d08edaf4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e68997931ede4b6398570a7d08edaf4d already in progress: creating tablet
I20251024 19:48:14.413528 1768 raft_consensus.cc:359] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.415282 1768 raft_consensus.cc:385] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.415261 1645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07e75c7f35544bec82cb994dea5ee4ef" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:14.415350 1768 raft_consensus.cc:740] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.415482 1645 raft_consensus.cc:2468] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:14.415611 1768 consensus_queue.cc:260] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.416983 1768 ts_tablet_manager.cc:1434] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20251024 19:48:14.417275 1768 tablet_bootstrap.cc:492] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:14.418300 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 715) took 2877 ms. Trace:
I20251024 19:48:14.418408 1776 rpcz_store.cc:276] 1024 19:48:11.540736 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.541147 (+ 411us) service_pool.cc:225] Handling call
1024 19:48:14.031784 (+2490637us) ts_tablet_manager.cc:1151] Deleting tablet 9b89a90bb4ca447fbaf34fe39001c261
1024 19:48:14.031801 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.233352 (+201551us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.239502 (+ 6150us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.418278 (+178776us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41235,"tablet-delete.queue_time_us":2490503}
I20251024 19:48:14.418911 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 720) took 2860 ms. Trace:
I20251024 19:48:14.422677 1768 tablet_bootstrap.cc:654] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.422623 2828 raft_consensus.cc:515] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:14.423244 2828 leader_election.cc:290] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:14.429219 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 607) took 5269 ms. Trace:
I20251024 19:48:14.429405 1775 rpcz_store.cc:276] 1024 19:48:09.159736 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.159846 (+ 110us) service_pool.cc:225] Handling call
1024 19:48:14.429066 (+5269220us) ts_tablet_manager.cc:1151] Deleting tablet 95ae017a2e084ccabfa69c9a5befe0e7
1024 19:48:14.429078 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.429194 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5232570}
W20251024 19:48:14.430100 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 95ae017a2e084ccabfa69c9a5befe0e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95ae017a2e084ccabfa69c9a5befe0e7 already in progress: creating tablet
I20251024 19:48:14.430590 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 609) took 5094 ms. Trace:
I20251024 19:48:14.430730 1775 rpcz_store.cc:276] 1024 19:48:09.336095 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.336678 (+ 583us) service_pool.cc:225] Handling call
1024 19:48:14.430484 (+5093806us) ts_tablet_manager.cc:1151] Deleting tablet 1e6a588092a44688bc96f16ac00e39f9
1024 19:48:14.430493 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.430573 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5076752}
W20251024 19:48:14.431242 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 1e6a588092a44688bc96f16ac00e39f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e6a588092a44688bc96f16ac00e39f9 already in progress: creating tablet
I20251024 19:48:14.431595 1775 tablet_replica.cc:333] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.431842 1775 raft_consensus.cc:2243] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:14.432070 1775 raft_consensus.cc:2272] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.432699 1775 ts_tablet_manager.cc:1916] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.439762 1767 tablet_bootstrap.cc:654] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.441143 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 9b89a90bb4ca447fbaf34fe39001c261 (table default.Table_f37eb464f5144d1a8135a5f1a44cee4f [id=2fda7fc5a71646b79f5d848dca570135]) successfully deleted
I20251024 19:48:14.448168 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b88a6a5237d146f5b6f082ca3af25b2b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:14.449004 1305 leader_election.cc:304] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:14.449323 2867 raft_consensus.cc:2749] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:14.457780 1775 ts_tablet_manager.cc:1929] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:14.459561 1775 log.cc:1199] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/59c7dbe9ff2c4869994c4f43d5676c30
I20251024 19:48:14.459936 1775 ts_tablet_manager.cc:1950] T 59c7dbe9ff2c4869994c4f43d5676c30 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.460749 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 611) took 5110 ms. Trace:
I20251024 19:48:14.461206 1775 rpcz_store.cc:276] 1024 19:48:09.350203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:09.350734 (+ 531us) service_pool.cc:225] Handling call
1024 19:48:14.431541 (+5080807us) ts_tablet_manager.cc:1151] Deleting tablet 59c7dbe9ff2c4869994c4f43d5676c30
1024 19:48:14.431553 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.444146 (+ 12593us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.457649 (+ 13503us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.460730 (+ 3081us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5062265}
W20251024 19:48:14.463898 1444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251024 19:48:14.464377 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 59c7dbe9ff2c4869994c4f43d5676c30 (table default.Table_eb9c8019451d46fba31a87a0b6ac6ad8 [id=e5b1dd31c2114e8a95ba33981029255c]) successfully deleted
I20251024 19:48:14.465709 2867 raft_consensus.cc:493] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:14.465864 2867 raft_consensus.cc:515] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.465983 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 637) took 4233 ms. Trace:
I20251024 19:48:14.466116 1775 rpcz_store.cc:276] 1024 19:48:10.232294 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.236616 (+ 4322us) service_pool.cc:225] Handling call
1024 19:48:14.465800 (+4229184us) ts_tablet_manager.cc:1151] Deleting tablet 64499c8734d34c66b0b52a2cd3ef0673
1024 19:48:14.465857 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.465967 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4219756}
I20251024 19:48:14.466630 2867 leader_election.cc:290] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:14.467067 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 64499c8734d34c66b0b52a2cd3ef0673 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64499c8734d34c66b0b52a2cd3ef0673 already in progress: creating tablet
I20251024 19:48:14.467540 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 639) took 4235 ms. Trace:
I20251024 19:48:14.467762 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "701c93a600964c20a275f5942a2ff863" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:14.467636 1775 rpcz_store.cc:276] 1024 19:48:10.232544 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.236676 (+ 4132us) service_pool.cc:225] Handling call
1024 19:48:14.467407 (+4230731us) ts_tablet_manager.cc:1151] Deleting tablet 6f89a3bbeb9b4d57a49078729110d951
1024 19:48:14.467418 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.467514 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4192215}
W20251024 19:48:14.468279 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 6f89a3bbeb9b4d57a49078729110d951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f89a3bbeb9b4d57a49078729110d951 already in progress: creating tablet
I20251024 19:48:14.468951 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "701c93a600964c20a275f5942a2ff863" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:14.469841 1305 leader_election.cc:304] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:14.470417 2867 raft_consensus.cc:2749] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:14.475657 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 635) took 4243 ms. Trace:
I20251024 19:48:14.475812 1775 rpcz_store.cc:276] 1024 19:48:10.232050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.236553 (+ 4503us) service_pool.cc:225] Handling call
1024 19:48:14.475521 (+4238968us) ts_tablet_manager.cc:1151] Deleting tablet 59956dad06064024ab171631ef6c1dca
1024 19:48:14.475533 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.475637 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4179974}
W20251024 19:48:14.476298 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 59956dad06064024ab171631ef6c1dca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59956dad06064024ab171631ef6c1dca already in progress: creating tablet
I20251024 19:48:14.476744 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 636) took 4244 ms. Trace:
I20251024 19:48:14.476864 1775 rpcz_store.cc:276] 1024 19:48:10.232152 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.236583 (+ 4431us) service_pool.cc:225] Handling call
1024 19:48:14.476600 (+4240017us) ts_tablet_manager.cc:1151] Deleting tablet d2c92975cf66441f9ac90c19772d3db3
1024 19:48:14.476608 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.476729 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4180847}
W20251024 19:48:14.477538 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet d2c92975cf66441f9ac90c19772d3db3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2c92975cf66441f9ac90c19772d3db3 already in progress: creating tablet
I20251024 19:48:14.477962 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 638) took 4245 ms. Trace:
I20251024 19:48:14.478080 1775 rpcz_store.cc:276] 1024 19:48:10.232448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.236647 (+ 4199us) service_pool.cc:225] Handling call
1024 19:48:14.477848 (+4241201us) ts_tablet_manager.cc:1151] Deleting tablet 2087e295ad4b4daa8eeebd49b43e08c8
1024 19:48:14.477859 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.477944 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4174450}
W20251024 19:48:14.478511 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2087e295ad4b4daa8eeebd49b43e08c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2087e295ad4b4daa8eeebd49b43e08c8 already in progress: creating tablet
I20251024 19:48:14.478825 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d38154d3ffd34f4b8d7305b1c23a4fae" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:14.478969 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 659) took 4029 ms. Trace:
I20251024 19:48:14.479090 1642 raft_consensus.cc:2393] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10a9536207e343f188e0ef709081d04e in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:14.479033 1775 rpcz_store.cc:276] 1024 19:48:10.449931 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.620682 (+170751us) service_pool.cc:225] Handling call
1024 19:48:14.478880 (+3858198us) ts_tablet_manager.cc:1151] Deleting tablet 2d39eb2f825b417a9361c6f4876e0cd7
1024 19:48:14.478888 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.478956 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3812900}
W20251024 19:48:14.479501 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2d39eb2f825b417a9361c6f4876e0cd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d39eb2f825b417a9361c6f4876e0cd7 already in progress: creating tablet
I20251024 19:48:14.479859 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 660) took 4029 ms. Trace:
I20251024 19:48:14.479964 1775 rpcz_store.cc:276] 1024 19:48:10.450048 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.666005 (+215957us) service_pool.cc:225] Handling call
1024 19:48:14.479767 (+3813762us) ts_tablet_manager.cc:1151] Deleting tablet c0f231f739b64f9fb6a726589a213146
1024 19:48:14.479776 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.479845 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3813621}
I20251024 19:48:14.480183 2877 tablet.cc:1717] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97: Skipping requested alter to schema version 3, tablet already version 3
W20251024 19:48:14.480386 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet c0f231f739b64f9fb6a726589a213146 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0f231f739b64f9fb6a726589a213146 already in progress: creating tablet
I20251024 19:48:14.480387 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c77f6bbf7b145389ca6cf74ddd3aa3f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:14.480571 1644 raft_consensus.cc:2468] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:14.480712 1775 tablet_replica.cc:333] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.480913 1775 raft_consensus.cc:2243] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:14.481241 1775 raft_consensus.cc:2272] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.481871 1775 ts_tablet_manager.cc:1916] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.482669 1444 leader_election.cc:304] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:14.483053 1444 leader_election.cc:304] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
W20251024 19:48:14.483232 1444 raft_consensus.cc:2686] T ce8c179e98944df0b510a0e8daf3920f P 10a9536207e343f188e0ef709081d04e: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251024 19:48:14.484216 1444 leader_election.cc:343] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 2037fcca87b94b5fb31639b4d07307b0
I20251024 19:48:14.484306 1444 leader_election.cc:304] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:14.484439 1444 raft_consensus.cc:2686] T 2037fcca87b94b5fb31639b4d07307b0 P 10a9536207e343f188e0ef709081d04e: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251024 19:48:14.485100 1444 leader_election.cc:304] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:14.485191 2808 raft_consensus.cc:2804] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:14.485253 2808 raft_consensus.cc:493] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:14.485297 2808 raft_consensus.cc:3060] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:14.485414 1444 consensus_peers.cc:597] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b89a90bb4ca447fbaf34fe39001c261. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:14.485690 2858 consensus_queue.cc:1048] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:14.486091 2858 raft_consensus.cc:2804] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:14.486157 2858 raft_consensus.cc:493] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:14.486219 2858 raft_consensus.cc:3060] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.487857 2808 raft_consensus.cc:515] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.488567 2857 consensus_queue.cc:1048] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:14.488708 2858 raft_consensus.cc:515] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.490228 2858 leader_election.cc:290] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:14.492648 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ffd446c269448d38cc8d987a53aecf9" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:14.492225 2615 consensus_queue.cc:1048] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:14.493041 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07e75c7f35544bec82cb994dea5ee4ef" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:14.492866 1444 leader_election.cc:304] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:14.493686 1447 leader_election.cc:343] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:14.493911 1447 leader_election.cc:343] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:14.494081 2857 raft_consensus.cc:3060] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.496636 2857 raft_consensus.cc:2749] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:14.496881 2808 leader_election.cc:290] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:14.493692 1444 leader_election.cc:304] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:14.497807 2858 raft_consensus.cc:2804] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:14.497886 2858 raft_consensus.cc:493] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:14.497948 2858 raft_consensus.cc:3060] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.500366 2858 raft_consensus.cc:515] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.501377 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c77f6bbf7b145389ca6cf74ddd3aa3f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:14.501863 1447 leader_election.cc:343] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:14.502892 2858 leader_election.cc:290] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:14.504379 2828 raft_consensus.cc:493] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:14.504583 2828 raft_consensus.cc:515] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.505323 2828 leader_election.cc:290] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:14.506007 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ffd446c269448d38cc8d987a53aecf9" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:14.506147 1644 raft_consensus.cc:3060] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.509733 1644 raft_consensus.cc:2468] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:14.510775 1775 ts_tablet_manager.cc:1929] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:14.511571 1775 log.cc:1199] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/fd52295b14b14e70bbdcb45f1bd849e4
I20251024 19:48:14.512122 1775 ts_tablet_manager.cc:1950] T fd52295b14b14e70bbdcb45f1bd849e4 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.515623 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 661) took 4065 ms. Trace:
I20251024 19:48:14.515803 1775 rpcz_store.cc:276] 1024 19:48:10.450133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.666143 (+216010us) service_pool.cc:225] Handling call
1024 19:48:14.480671 (+3814528us) ts_tablet_manager.cc:1151] Deleting tablet fd52295b14b14e70bbdcb45f1bd849e4
1024 19:48:14.480679 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.505403 (+ 24724us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.510672 (+ 5269us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.515601 (+ 4929us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3814406}
I20251024 19:48:14.516988 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet fd52295b14b14e70bbdcb45f1bd849e4 (table default.Table_ff7a43292b9045f9bbfa71883bcd7e61 [id=e2c539db2aa148448925a69f4848feb2]) successfully deleted
I20251024 19:48:14.517478 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 662) took 4067 ms. Trace:
I20251024 19:48:14.517624 1775 rpcz_store.cc:276] 1024 19:48:10.474928 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.666270 (+191342us) service_pool.cc:225] Handling call
1024 19:48:14.517347 (+3851077us) ts_tablet_manager.cc:1151] Deleting tablet 7be9290daabb404f955f05922d897fbf
1024 19:48:14.517358 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.517459 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3850980}
W20251024 19:48:14.518142 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7be9290daabb404f955f05922d897fbf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7be9290daabb404f955f05922d897fbf already in progress: creating tablet
I20251024 19:48:14.518534 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 663) took 4043 ms. Trace:
I20251024 19:48:14.518656 1775 rpcz_store.cc:276] 1024 19:48:10.475087 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.666369 (+191282us) service_pool.cc:225] Handling call
1024 19:48:14.518429 (+3852060us) ts_tablet_manager.cc:1151] Deleting tablet 7e0dc7d4fed5448cb616af29e405e766
1024 19:48:14.518438 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.518516 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3851949}
W20251024 19:48:14.519138 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7e0dc7d4fed5448cb616af29e405e766 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e0dc7d4fed5448cb616af29e405e766 already in progress: creating tablet
I20251024 19:48:14.519515 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 668) took 3968 ms. Trace:
I20251024 19:48:14.519631 1775 rpcz_store.cc:276] 1024 19:48:10.551386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.720184 (+168798us) service_pool.cc:225] Handling call
1024 19:48:14.519415 (+3799231us) ts_tablet_manager.cc:1151] Deleting tablet ae0d26f010bd46d3a05ac50f966d5769
1024 19:48:14.519425 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.519498 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3771426}
W20251024 19:48:14.520058 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet ae0d26f010bd46d3a05ac50f966d5769 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae0d26f010bd46d3a05ac50f966d5769 already in progress: creating tablet
I20251024 19:48:14.418996 1776 rpcz_store.cc:276] 1024 19:48:11.558785 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.560476 (+ 1691us) service_pool.cc:225] Handling call
1024 19:48:14.418762 (+2858286us) ts_tablet_manager.cc:1151] Deleting tablet 82cf75ee148e45b29da3343b3b068781
1024 19:48:14.418775 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.418890 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2858060}
I20251024 19:48:14.520390 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 669) took 3968 ms. Trace:
I20251024 19:48:14.520529 1775 rpcz_store.cc:276] 1024 19:48:10.552194 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.720731 (+168537us) service_pool.cc:225] Handling call
1024 19:48:14.520300 (+3799569us) ts_tablet_manager.cc:1151] Deleting tablet 55aa786b9f6847baa265802cc28b2b80
1024 19:48:14.520308 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.520375 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3772189}
W20251024 19:48:14.520757 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 82cf75ee148e45b29da3343b3b068781 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82cf75ee148e45b29da3343b3b068781 already in progress: creating tablet
W20251024 19:48:14.520982 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 55aa786b9f6847baa265802cc28b2b80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55aa786b9f6847baa265802cc28b2b80 already in progress: creating tablet
I20251024 19:48:14.521311 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 670) took 3955 ms. Trace:
I20251024 19:48:14.521426 1775 rpcz_store.cc:276] 1024 19:48:10.565816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.721498 (+155682us) service_pool.cc:225] Handling call
1024 19:48:14.521198 (+3799700us) ts_tablet_manager.cc:1151] Deleting tablet 27f1a5eccdf6430c9e220988043596b0
1024 19:48:14.521207 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.521297 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3772974}
W20251024 19:48:14.521886 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 27f1a5eccdf6430c9e220988043596b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27f1a5eccdf6430c9e220988043596b0 already in progress: creating tablet
I20251024 19:48:14.522230 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 671) took 3914 ms. Trace:
I20251024 19:48:14.522333 1775 rpcz_store.cc:276] 1024 19:48:10.607467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.722152 (+114685us) service_pool.cc:225] Handling call
1024 19:48:14.522142 (+3799990us) ts_tablet_manager.cc:1151] Deleting tablet 4ef1ab2b2d2c41f09c96c895942ef743
1024 19:48:14.522150 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.522216 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3773869}
W20251024 19:48:14.522789 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4ef1ab2b2d2c41f09c96c895942ef743 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ef1ab2b2d2c41f09c96c895942ef743 already in progress: creating tablet
I20251024 19:48:14.523247 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 667) took 3973 ms. Trace:
I20251024 19:48:14.523347 1775 rpcz_store.cc:276] 1024 19:48:10.549876 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:10.719866 (+169990us) service_pool.cc:225] Handling call
1024 19:48:14.523158 (+3803292us) ts_tablet_manager.cc:1151] Deleting tablet 069091f1c238487580baab38af6a70e2
1024 19:48:14.523166 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.523232 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3762070}
I20251024 19:48:14.523684 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07e75c7f35544bec82cb994dea5ee4ef" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
W20251024 19:48:14.523797 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 069091f1c238487580baab38af6a70e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 069091f1c238487580baab38af6a70e2 already in progress: creating tablet
I20251024 19:48:14.523900 1649 raft_consensus.cc:3060] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.524161 1775 tablet_replica.cc:333] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.524524 1775 raft_consensus.cc:2243] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:14.524744 1775 raft_consensus.cc:2272] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.525681 1775 ts_tablet_manager.cc:1916] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.527490 1649 raft_consensus.cc:2468] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:14.529500 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c77f6bbf7b145389ca6cf74ddd3aa3f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:14.529743 1643 raft_consensus.cc:3060] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.532305 1775 ts_tablet_manager.cc:1929] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:14.532435 1775 log.cc:1199] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/d9bf08ccf7054fedb9efcbb00354c71a
I20251024 19:48:14.532829 1775 ts_tablet_manager.cc:1950] T d9bf08ccf7054fedb9efcbb00354c71a P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.533245 1643 raft_consensus.cc:2468] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:14.533937 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 675) took 3492 ms. Trace:
I20251024 19:48:14.534227 1775 rpcz_store.cc:276] 1024 19:48:11.041880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.041940 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:14.524073 (+3482133us) ts_tablet_manager.cc:1151] Deleting tablet d9bf08ccf7054fedb9efcbb00354c71a
1024 19:48:14.524081 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.529174 (+ 5093us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.532140 (+ 2966us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.533892 (+ 1752us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3481949}
I20251024 19:48:14.534699 1444 leader_election.cc:304] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:14.535260 1444 leader_election.cc:304] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:14.535678 1444 leader_election.cc:304] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:14.536046 2858 raft_consensus.cc:2804] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:14.536202 2858 raft_consensus.cc:697] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:14.536191 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 723) took 2969 ms. Trace:
I20251024 19:48:14.536285 1776 rpcz_store.cc:276] 1024 19:48:11.567170 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.571012 (+ 3842us) service_pool.cc:225] Handling call
1024 19:48:14.536028 (+2965016us) ts_tablet_manager.cc:1151] Deleting tablet 8876b7b38e744212a3722dbe5bcc5596
1024 19:48:14.536043 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.536169 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2964836}
I20251024 19:48:14.536473 2858 consensus_queue.cc:237] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:14.536824 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 8876b7b38e744212a3722dbe5bcc5596 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8876b7b38e744212a3722dbe5bcc5596 already in progress: creating tablet
I20251024 19:48:14.537269 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 725) took 2953 ms. Trace:
I20251024 19:48:14.537351 1776 rpcz_store.cc:276] 1024 19:48:11.583350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.583936 (+ 586us) service_pool.cc:225] Handling call
1024 19:48:14.537137 (+2953201us) ts_tablet_manager.cc:1151] Deleting tablet cb0405765c6146cbbb187af96a60be2e
1024 19:48:14.537150 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.537250 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2952942}
I20251024 19:48:14.537552 2808 raft_consensus.cc:2804] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:14.537720 2808 raft_consensus.cc:697] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
W20251024 19:48:14.537894 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet cb0405765c6146cbbb187af96a60be2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb0405765c6146cbbb187af96a60be2e already in progress: creating tablet
I20251024 19:48:14.537972 2808 consensus_queue.cc:237] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.538476 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 729) took 2866 ms. Trace:
I20251024 19:48:14.538556 1776 rpcz_store.cc:276] 1024 19:48:11.671835 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.672194 (+ 359us) service_pool.cc:225] Handling call
1024 19:48:14.538358 (+2866164us) ts_tablet_manager.cc:1151] Deleting tablet 50cea450eda9485daba40bf875a1b90c
1024 19:48:14.538370 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.538459 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2865866}
I20251024 19:48:14.538834 2857 raft_consensus.cc:2804] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
W20251024 19:48:14.538964 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 50cea450eda9485daba40bf875a1b90c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50cea450eda9485daba40bf875a1b90c already in progress: creating tablet
I20251024 19:48:14.542687 1372 raft_consensus.cc:3060] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.542989 1557 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
W20251024 19:48:14.543934 1629 consensus_meta.cc:287] T fc35c0a2b36c45ffa2fe4ad00abccfd1 P af3952de050140d383f6b0de7666fa97: Time spent flushing consensus metadata: real 0.772s user 0.021s sys 0.000s
I20251024 19:48:14.549082 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet d9bf08ccf7054fedb9efcbb00354c71a (table default.Table_4a085504a0884b37b6787b3bcf7a847f [id=10833938cf88466b834698420d7c01e3]) successfully deleted
I20251024 19:48:14.549963 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 674) took 3508 ms. Trace:
I20251024 19:48:14.550482 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet ad81d80697984ff1834a7d795e0f4b9b from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:14.550405 2857 raft_consensus.cc:697] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:14.551182 1183 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 187925082cdd4923a6e9148bf9f270fd (table default.Table_55f8c66bd97343438051a98af12d8c2c [id=33f5ac73298543d0b5af77a27e897b79]). Expected version 1 got 0
I20251024 19:48:14.551158 2857 consensus_queue.cc:237] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.551529 1183 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet e29abfa3bc8944aea7d1acd03e3eb897 (table default.Table_55339e02660843c3a8536a4b7a06831e [id=ba30503223dd4979824ec9c1a50a8ae2]). Expected version 1 got 0
I20251024 19:48:14.553800 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet 2346c03802a245a2af8c086b626d660d from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:14.553862 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f37a01dc3d5f458d81605f2c41fa27dd consensus state has no leader
I20251024 19:48:14.554183 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet 839cef50795840fab564b94d88565f8c from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:14.557718 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 9ab081e3a2b2445f8b8c73d271416cce (DEFAULT_TABLE table=default.Table_542a694472c54365a9cc9ae4d484013e [id=dd3044d4ed3047e8900972e8d2fddf88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.558125 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ab081e3a2b2445f8b8c73d271416cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.558421 1766 tablet_bootstrap.cc:492] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:14.558485 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 73ab749795a64339abeaf1ce5d6570ab (DEFAULT_TABLE table=default.Table_86e47213e04046e89c4ef6974b4b3630 [id=619ed3392c9940d2b1a9856cb417f384]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.558590 1766 ts_tablet_manager.cc:1403] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.324s user 0.052s sys 0.078s
I20251024 19:48:14.557761 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 8942872216024d1e94a99643ed4974ed (DEFAULT_TABLE table=default.Table_a8a6e3b550974b0e8267a1e5f3d769e5 [id=70269607f68449c7b2e76063026369f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.559342 1766 raft_consensus.cc:359] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.559546 1766 raft_consensus.cc:385] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.559612 1766 raft_consensus.cc:740] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.559846 1766 consensus_queue.cc:260] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.560411 1766 ts_tablet_manager.cc:1434] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:14.560397 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 9ab081e3a2b2445f8b8c73d271416cce (DEFAULT_TABLE table=default.Table_542a694472c54365a9cc9ae4d484013e [id=dd3044d4ed3047e8900972e8d2fddf88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.560734 1766 tablet_bootstrap.cc:492] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:14.560782 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ab081e3a2b2445f8b8c73d271416cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.561815 1344 tablet_service.cc:1505] Processing CreateTablet for tablet ce9ad86d489a4df1a7b687514c4d1512 (DEFAULT_TABLE table=default.Table_a157b3f500ac43c285d1b0c80f0bd88b [id=542e76acc5d8498192718ff9a4fa1837]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.562296 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 73ab749795a64339abeaf1ce5d6570ab (DEFAULT_TABLE table=default.Table_86e47213e04046e89c4ef6974b4b3630 [id=619ed3392c9940d2b1a9856cb417f384]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.562641 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ab749795a64339abeaf1ce5d6570ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.562709 1766 tablet_bootstrap.cc:654] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.565768 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet fd52295b14b14e70bbdcb45f1bd849e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.566118 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 8942872216024d1e94a99643ed4974ed (DEFAULT_TABLE table=default.Table_a8a6e3b550974b0e8267a1e5f3d769e5 [id=70269607f68449c7b2e76063026369f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.566447 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8942872216024d1e94a99643ed4974ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.569705 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 187bc968c2b049028dd25dfb0264c0aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.570683 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1dbcae61809349c59e959bff6c5cdf24" } modify_external_catalogs: true
I20251024 19:48:14.570930 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1dbcae61809349c59e959bff6c5cdf24" } modify_external_catalogs: true
I20251024 19:48:14.572817 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2021df09852e4dd09dbf7b36a0a3ef5b" } modify_external_catalogs: true
I20251024 19:48:14.573772 1766 tablet_bootstrap.cc:492] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:14.573920 1766 ts_tablet_manager.cc:1403] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20251024 19:48:14.574252 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a1c0c56a2bd5452293c935a6aecb39e3" } modify_external_catalogs: true
I20251024 19:48:14.574419 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a1c0c56a2bd5452293c935a6aecb39e3" } modify_external_catalogs: true
I20251024 19:48:14.574704 1766 raft_consensus.cc:359] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.574955 1766 raft_consensus.cc:385] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.575057 1766 raft_consensus.cc:740] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.575014 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fc2a1aeddc444f578693d922323862f5" } modify_external_catalogs: true
I20251024 19:48:14.575188 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fc2a1aeddc444f578693d922323862f5" } modify_external_catalogs: true
I20251024 19:48:14.573930 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2021df09852e4dd09dbf7b36a0a3ef5b" } modify_external_catalogs: true
I20251024 19:48:14.573638 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7fd523f97f3e4c549d353c15d2ac6e36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:14.577113 1766 consensus_queue.cc:260] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:14.577596 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fd523f97f3e4c549d353c15d2ac6e36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:14.578054 1766 ts_tablet_manager.cc:1434] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251024 19:48:14.578341 1766 tablet_bootstrap.cc:492] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:14.579962 1766 tablet_bootstrap.cc:654] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:14.579950 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f0b1e072db9742c38ce3b09bd195dcc0 consensus state has no leader
I20251024 19:48:14.580351 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e7975a91f4674c4fa7904e3b7bc780dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:14.580888 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7975a91f4674c4fa7904e3b7bc780dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:14.585170 1766 tablet_bootstrap.cc:492] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:14.585326 1766 ts_tablet_manager.cc:1403] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20251024 19:48:14.586210 1766 raft_consensus.cc:359] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.586423 1766 raft_consensus.cc:385] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.586512 1766 raft_consensus.cc:740] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.588550 1766 consensus_queue.cc:260] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.589008 1474 tablet_service.cc:1505] Processing CreateTablet for tablet ce9ad86d489a4df1a7b687514c4d1512 (DEFAULT_TABLE table=default.Table_a157b3f500ac43c285d1b0c80f0bd88b [id=542e76acc5d8498192718ff9a4fa1837]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.589419 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce9ad86d489a4df1a7b687514c4d1512. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.589386 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 643) took 3202 ms. Trace:
I20251024 19:48:14.589529 1774 rpcz_store.cc:276] 1024 19:48:11.386693 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.386882 (+ 189us) service_pool.cc:225] Handling call
1024 19:48:13.893639 (+2506757us) ts_tablet_manager.cc:1151] Deleting tablet 73ffe342f541447d8519ee24bc98a0f3
1024 19:48:13.893655 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.133227 (+239572us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.136999 (+ 3772us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.589364 (+452365us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2506571}
I20251024 19:48:14.589824 1766 ts_tablet_manager.cc:1434] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251024 19:48:14.590027 1774 tablet_replica.cc:333] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:14.590142 1766 tablet_bootstrap.cc:492] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:14.590318 1774 raft_consensus.cc:2243] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:14.590513 1774 raft_consensus.cc:2272] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.591112 1774 ts_tablet_manager.cc:1916] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.594359 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9e7fa6339ee24bab9d78fd7c318ed78d" } modify_external_catalogs: true
I20251024 19:48:14.594624 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9e7fa6339ee24bab9d78fd7c318ed78d" } modify_external_catalogs: true
I20251024 19:48:14.594782 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3abf70710bce4728bfecd4a89d31dea6" } new_table_name: "default.Table_ab448d5388b84b3489200f3a17c24edb" modify_external_catalogs: true
I20251024 19:48:14.599375 1337 tablet_service.cc:1505] Processing CreateTablet for tablet b6803d23a4714934b85190886611d8d1 (DEFAULT_TABLE table=default.Table_c7859afca8534349935391ebab856798 [id=5abc2c497d6444cfb75eb3a2498c689d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.599889 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 605205170c994296a5ef0dc568788517 (DEFAULT_TABLE table=default.Table_f518918f7bfe409fae8f42ac3d059872 [id=99337e2056294b30ad381de2496412a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.600845 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 73ffe342f541447d8519ee24bc98a0f3 (table default.Table_74e4f24f12db403686548924c83c5191 [id=508d6cdc5d4a494bad31eaf36383e214]) successfully deleted
I20251024 19:48:14.601292 2838 raft_consensus.cc:2749] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:14.609037 1349 tablet_service.cc:1505] Processing CreateTablet for tablet d933103af272429c95f8d3fc54e427da (DEFAULT_TABLE table=default.Table_f75802f81eff41248b27d50a04c4efbc [id=cc417c0457034d709f36cbac1f56179c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.610813 1340 tablet_service.cc:1505] Processing CreateTablet for tablet eaafc2aff3244c8e8aa7bbe4aa59cb7c (DEFAULT_TABLE table=default.Table_485cf0780a4c4dfe809deb6fb31a4671 [id=b29d365ff77541b88ae8cbdec98dea30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.614887 1775 rpcz_store.cc:276] 1024 19:48:11.041786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.045210 (+ 3424us) service_pool.cc:225] Handling call
1024 19:48:14.549821 (+3504611us) ts_tablet_manager.cc:1151] Deleting tablet dc1b6f0a377945089c50012f74d38564
1024 19:48:14.549834 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.549915 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3504421}
I20251024 19:48:14.615952 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 681) took 3431 ms. Trace:
I20251024 19:48:14.616176 1775 rpcz_store.cc:276] 1024 19:48:11.184391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.184654 (+ 263us) service_pool.cc:225] Handling call
1024 19:48:14.615597 (+3430943us) ts_tablet_manager.cc:1151] Deleting tablet 3da81c18d9264f38966944ad13f256d8
1024 19:48:14.615648 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.615898 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3430709}
I20251024 19:48:14.617771 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 682) took 3400 ms. Trace:
I20251024 19:48:14.618203 1775 rpcz_store.cc:276] 1024 19:48:11.217686 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.217936 (+ 250us) service_pool.cc:225] Handling call
1024 19:48:14.617138 (+3399202us) ts_tablet_manager.cc:1151] Deleting tablet 250f106664be4b5aaa330dbb4e43cb7b
1024 19:48:14.617151 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.617634 (+ 483us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3398889}
W20251024 19:48:14.621351 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet dc1b6f0a377945089c50012f74d38564 because the tablet was not found. No further retry: Not found: Tablet not found: dc1b6f0a377945089c50012f74d38564
W20251024 19:48:14.621652 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3da81c18d9264f38966944ad13f256d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3da81c18d9264f38966944ad13f256d8 already in progress: creating tablet
W20251024 19:48:14.621909 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 250f106664be4b5aaa330dbb4e43cb7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 250f106664be4b5aaa330dbb4e43cb7b already in progress: creating tablet
I20251024 19:48:14.623261 1488 tablet_service.cc:1505] Processing CreateTablet for tablet b6803d23a4714934b85190886611d8d1 (DEFAULT_TABLE table=default.Table_c7859afca8534349935391ebab856798 [id=5abc2c497d6444cfb75eb3a2498c689d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.625931 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6803d23a4714934b85190886611d8d1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:14.634995 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 489752031a5a4d168367cb4d143702b7 consensus state has no leader
I20251024 19:48:14.636989 1766 tablet_bootstrap.cc:654] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.629426 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 605205170c994296a5ef0dc568788517 (DEFAULT_TABLE table=default.Table_f518918f7bfe409fae8f42ac3d059872 [id=99337e2056294b30ad381de2496412a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.630242 1474 tablet_service.cc:1505] Processing CreateTablet for tablet d933103af272429c95f8d3fc54e427da (DEFAULT_TABLE table=default.Table_f75802f81eff41248b27d50a04c4efbc [id=cc417c0457034d709f36cbac1f56179c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.641628 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d933103af272429c95f8d3fc54e427da. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:14.642109 1178 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 5ace18d554f14acc892bc956c30e9979 (table default.Table_9b49c126d44b4afeb3d1122999d2b8da [id=a2f87693fc294ee5a7b0ec6d386c9b25], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5ace18d554f14acc892bc956c30e9979 consensus state has no leader
I20251024 19:48:14.642856 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 605205170c994296a5ef0dc568788517. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.647454 1766 tablet_bootstrap.cc:492] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:14.648054 1766 ts_tablet_manager.cc:1403] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.058s user 0.026s sys 0.002s
I20251024 19:48:14.647491 2858 raft_consensus.cc:493] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:14.648368 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_177f4c4e8d3b437aa14bb9e8a383510a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:14.648947 2838 raft_consensus.cc:493] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251024 19:48:14.649024 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_177f4c4e8d3b437aa14bb9e8a383510a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:14.649086 2838 raft_consensus.cc:515] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.650050 2838 leader_election.cc:290] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:14.648571 2858 raft_consensus.cc:515] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:14.653801 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505e4fb0755a4caaa462e5f5154bfee2" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:14.653614 2858 leader_election.cc:290] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:14.654294 1447 leader_election.cc:343] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:14.655665 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b3e9960b7821421aa23e7ef5389f894a
I20251024 19:48:14.656194 1189 catalog_manager.cc:6719] ReplaceTablet: tablet b3e9960b7821421aa23e7ef5389f894a deleted and replaced by tablet 4f10340e106b473eb715b74b129835b5
I20251024 19:48:14.657624 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet b3e9960b7821421aa23e7ef5389f894a with delete_type TABLET_DATA_DELETED (replaced by tablet 4f10340e106b473eb715b74b129835b5) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:14.658538 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet b3e9960b7821421aa23e7ef5389f894a with delete_type TABLET_DATA_DELETED (replaced by tablet 4f10340e106b473eb715b74b129835b5) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.663535 1189 catalog_manager.cc:5649] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:14.664024 1189 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 4ffd446c269448d38cc8d987a53aecf9 (table default.Table_bedd8edc24014953ac8c7dcecf18f128 [id=2ae1c808837e48df97c4efb01a684da9]). Expected version 1 got 0
I20251024 19:48:14.648319 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 684) took 3427 ms. Trace:
I20251024 19:48:14.650029 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "635a132735704cfda69bf7c8deaea81d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:14.665232 1775 rpcz_store.cc:276] 1024 19:48:11.220480 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.220524 (+ 44us) service_pool.cc:225] Handling call
1024 19:48:14.646884 (+3426360us) ts_tablet_manager.cc:1151] Deleting tablet 28005301f9de48e5826f46c79aeb5fe5
1024 19:48:14.647213 (+ 329us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.648250 (+ 1037us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3425750}
W20251024 19:48:14.667718 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 28005301f9de48e5826f46c79aeb5fe5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28005301f9de48e5826f46c79aeb5fe5 already in progress: creating tablet
I20251024 19:48:14.666203 1482 tablet_service.cc:1505] Processing CreateTablet for tablet eaafc2aff3244c8e8aa7bbe4aa59cb7c (DEFAULT_TABLE table=default.Table_485cf0780a4c4dfe809deb6fb31a4671 [id=b29d365ff77541b88ae8cbdec98dea30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.669764 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaafc2aff3244c8e8aa7bbe4aa59cb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.670136 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 3e09c12bcfd14ab081ab309f9dec4afc (DEFAULT_TABLE table=default.Table_f145f64b26134e0e9fa7c293d5d7842f [id=378d8732b5a24c9899ca74c62d6e88d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.669083 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 683) took 3448 ms. Trace:
I20251024 19:48:14.670328 1775 rpcz_store.cc:276] 1024 19:48:11.220362 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.220748 (+ 386us) service_pool.cc:225] Handling call
1024 19:48:14.668014 (+3447266us) ts_tablet_manager.cc:1151] Deleting tablet 23a1b73314084848a4f28a042612f394
1024 19:48:14.668281 (+ 267us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.669059 (+ 778us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3446975}
W20251024 19:48:14.672015 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 23a1b73314084848a4f28a042612f394 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23a1b73314084848a4f28a042612f394 already in progress: creating tablet
I20251024 19:48:14.672731 1775 tablet_replica.cc:333] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:14.674612 1775 raft_consensus.cc:2243] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:14.677767 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 3e09c12bcfd14ab081ab309f9dec4afc (DEFAULT_TABLE table=default.Table_f145f64b26134e0e9fa7c293d5d7842f [id=378d8732b5a24c9899ca74c62d6e88d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.678174 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e09c12bcfd14ab081ab309f9dec4afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.648972 1766 raft_consensus.cc:359] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.684767 1766 raft_consensus.cc:385] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.684890 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet e51e597b8fd044e393a3af07da25586c
I20251024 19:48:14.684929 1775 raft_consensus.cc:2272] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:14.686251 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9684072368f14cc2942b4e240f90b0f2
I20251024 19:48:14.686272 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet e51e597b8fd044e393a3af07da25586c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:14.687892 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet e51e597b8fd044e393a3af07da25586c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.688409 1338 tablet_service.cc:1505] Processing CreateTablet for tablet c1ca0dfa7e4c4f0fb59fe2bb8fa192ad (DEFAULT_TABLE table=default.Table_1ece0128472c4ef2a4e5f261fe73b3a1 [id=c73b9c050bd94f30bd0a665bed411295]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.685045 1766 raft_consensus.cc:740] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.689474 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 9684072368f14cc2942b4e240f90b0f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:14.691066 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 9684072368f14cc2942b4e240f90b0f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.693128 1370 raft_consensus.cc:3060] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.693086 2864 raft_consensus.cc:493] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af3952de050140d383f6b0de7666fa97)
I20251024 19:48:14.693322 2864 raft_consensus.cc:515] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.694258 2864 leader_election.cc:290] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:14.546577 2625 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.000s sys 0.020s
W20251024 19:48:14.698101 2625 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.903s user 0.000s sys 0.020s
I20251024 19:48:14.555220 1627 tablet_service.cc:1505] Processing CreateTablet for tablet 8942872216024d1e94a99643ed4974ed (DEFAULT_TABLE table=default.Table_a8a6e3b550974b0e8267a1e5f3d769e5 [id=70269607f68449c7b2e76063026369f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.699477 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8942872216024d1e94a99643ed4974ed. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:14.555430 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 198992us
I20251024 19:48:14.707049 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e29abfa3bc8944aea7d1acd03e3eb897" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:14.556434 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 73ab749795a64339abeaf1ce5d6570ab (DEFAULT_TABLE table=default.Table_86e47213e04046e89c4ef6974b4b3630 [id=619ed3392c9940d2b1a9856cb417f384]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.710008 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ab749795a64339abeaf1ce5d6570ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.583879 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 9ab081e3a2b2445f8b8c73d271416cce (DEFAULT_TABLE table=default.Table_542a694472c54365a9cc9ae4d484013e [id=dd3044d4ed3047e8900972e8d2fddf88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.584455 1616 tablet_service.cc:1505] Processing CreateTablet for tablet ce9ad86d489a4df1a7b687514c4d1512 (DEFAULT_TABLE table=default.Table_a157b3f500ac43c285d1b0c80f0bd88b [id=542e76acc5d8498192718ff9a4fa1837]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.588869 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 605205170c994296a5ef0dc568788517 (DEFAULT_TABLE table=default.Table_f518918f7bfe409fae8f42ac3d059872 [id=99337e2056294b30ad381de2496412a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.614609 1622 tablet_service.cc:1505] Processing CreateTablet for tablet d933103af272429c95f8d3fc54e427da (DEFAULT_TABLE table=default.Table_f75802f81eff41248b27d50a04c4efbc [id=cc417c0457034d709f36cbac1f56179c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.617750 1624 tablet_service.cc:1505] Processing CreateTablet for tablet eaafc2aff3244c8e8aa7bbe4aa59cb7c (DEFAULT_TABLE table=default.Table_485cf0780a4c4dfe809deb6fb31a4671 [id=b29d365ff77541b88ae8cbdec98dea30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.624666 1623 tablet_service.cc:1505] Processing CreateTablet for tablet b6803d23a4714934b85190886611d8d1 (DEFAULT_TABLE table=default.Table_c7859afca8534349935391ebab856798 [id=5abc2c497d6444cfb75eb3a2498c689d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.625461 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 727) took 2956 ms. Trace:
I20251024 19:48:14.651072 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "635a132735704cfda69bf7c8deaea81d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:14.654168 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505e4fb0755a4caaa462e5f5154bfee2" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:14.714745 1305 leader_election.cc:304] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:14.715317 2864 raft_consensus.cc:2749] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:14.656447 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet b3e9960b7821421aa23e7ef5389f894a with delete_type TABLET_DATA_DELETED (replaced by tablet 4f10340e106b473eb715b74b129835b5) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.717372 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 605205170c994296a5ef0dc568788517. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.717872 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7cab9c17496947efbbe3cd9e5d8a49a8
I20251024 19:48:14.718677 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d933103af272429c95f8d3fc54e427da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.719290 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaafc2aff3244c8e8aa7bbe4aa59cb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.719411 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 7cab9c17496947efbbe3cd9e5d8a49a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:14.720043 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce9ad86d489a4df1a7b687514c4d1512. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.720896 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 292c31e7d4ce411bb336a1405ca43ba5
I20251024 19:48:14.722185 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 292c31e7d4ce411bb336a1405ca43ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:14.722301 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 292c31e7d4ce411bb336a1405ca43ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.722855 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 7cab9c17496947efbbe3cd9e5d8a49a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.694909 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b32126d04e480f8db7b2b20aba5873" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:14.705336 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d38154d3ffd34f4b8d7305b1c23a4fae" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:14.718916 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 7cab9c17496947efbbe3cd9e5d8a49a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.725533 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 292c31e7d4ce411bb336a1405ca43ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.749949 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4b8dc796eac74a329a4e65bbf0410b3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:14.750697 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b8dc796eac74a329a4e65bbf0410b3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:14.750985 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf241a9566c34b7b911b69aaf4de06a2 consensus state has no leader
I20251024 19:48:14.751480 1508 raft_consensus.cc:2468] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:14.692780 1766 consensus_queue.cc:260] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.716470 1776 rpcz_store.cc:276] 1024 19:48:11.669159 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.672536 (+ 3377us) service_pool.cc:225] Handling call
1024 19:48:14.625288 (+2952752us) ts_tablet_manager.cc:1151] Deleting tablet cbd321c059e34834a6257fab5e6d14bb
1024 19:48:14.625303 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.625442 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2952534}
W20251024 19:48:14.758961 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet cbd321c059e34834a6257fab5e6d14bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbd321c059e34834a6257fab5e6d14bb already in progress: creating tablet
I20251024 19:48:14.759707 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 728) took 3088 ms. Trace:
I20251024 19:48:14.760598 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a6498b82a207414dafe80255fff08e8c" } new_table_name: "default.Table_f30acab987914de4a2372f8aeec08030" modify_external_catalogs: true
I20251024 19:48:14.761234 1766 ts_tablet_manager.cc:1434] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.113s user 0.001s sys 0.001s
I20251024 19:48:14.761937 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 187925082cdd4923a6e9148bf9f270fd (table default.Table_55f8c66bd97343438051a98af12d8c2c [id=33f5ac73298543d0b5af77a27e897b79]). Expected version 1 got 0
I20251024 19:48:14.762174 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 03cdbd1b62b741249ad78562a7edfd22 (table default.Table_42c3a4a03db14d0684721ce6af6960c2 [id=ca742ae585e0406abf43193a23aa9731]). Expected version 2 got 0
I20251024 19:48:14.763664 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_04c43324857b4aa7b8a4c4f6eb0769f3" } new_table_name: "default.Table_54107beea0ca41649b27ed9402844bd2" modify_external_catalogs: true
I20251024 19:48:14.764225 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 27b32126d04e480f8db7b2b20aba5873 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:14.764544 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b2be2ca906f34c9c855f39abd4548bf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:14.764676 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_dbfe837064fb41238410f42ada6b39a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:14.765259 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbfe837064fb41238410f42ada6b39a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:14.764241 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b89a90bb4ca447fbaf34fe39001c261 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:14.765257 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2be2ca906f34c9c855f39abd4548bf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:14.768812 2776 log.cc:927] Time spent T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20251024 19:48:14.772536 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_01c58de0bf6845b08ff6fab88827cdec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:14.772786 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d6276fea31c74f33b4f8c1258ec7b0a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:14.773294 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6276fea31c74f33b4f8c1258ec7b0a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:14.773813 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01c58de0bf6845b08ff6fab88827cdec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:14.774009 1477 tablet_service.cc:1505] Processing CreateTablet for tablet c1ca0dfa7e4c4f0fb59fe2bb8fa192ad (DEFAULT_TABLE table=default.Table_1ece0128472c4ef2a4e5f261fe73b3a1 [id=c73b9c050bd94f30bd0a665bed411295]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.774410 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1ca0dfa7e4c4f0fb59fe2bb8fa192ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.744374 1775 ts_tablet_manager.cc:1916] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:14.706970 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e29abfa3bc8944aea7d1acd03e3eb897" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:14.781190 1510 raft_consensus.cc:2468] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:14.784583 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 520060a9adca4bb0b8616bcc006dce52
I20251024 19:48:14.784623 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 529580599bfa4800973b8540ab13d22a (DEFAULT_TABLE table=default.Table_80225fc48e434efc981690a21c61cd00 [id=df862db2f61a4bc8a2dbdeae0e9873ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.785370 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 520060a9adca4bb0b8616bcc006dce52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:14.786021 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 520060a9adca4bb0b8616bcc006dce52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.788008 1776 rpcz_store.cc:276] 1024 19:48:11.671658 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.673350 (+ 1692us) service_pool.cc:225] Handling call
1024 19:48:14.759499 (+3086149us) ts_tablet_manager.cc:1151] Deleting tablet 6f7b6d15f25d4c2a905da6597ba8d98a
1024 19:48:14.759516 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.759682 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085945}
I20251024 19:48:14.789553 1766 tablet_bootstrap.cc:492] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:14.688216 2569 raft_consensus.cc:493] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:14.790148 2569 raft_consensus.cc:515] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:14.790479 1643 raft_consensus.cc:2468] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:14.791384 2569 leader_election.cc:290] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:14.793654 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 529580599bfa4800973b8540ab13d22a (DEFAULT_TABLE table=default.Table_80225fc48e434efc981690a21c61cd00 [id=df862db2f61a4bc8a2dbdeae0e9873ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.793821 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 529580599bfa4800973b8540ab13d22a (DEFAULT_TABLE table=default.Table_80225fc48e434efc981690a21c61cd00 [id=df862db2f61a4bc8a2dbdeae0e9873ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.794320 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 529580599bfa4800973b8540ab13d22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.688364 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet e51e597b8fd044e393a3af07da25586c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.794835 1444 leader_election.cc:304] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:14.795231 2808 raft_consensus.cc:2804] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:14.795300 2808 raft_consensus.cc:493] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:14.795372 2808 raft_consensus.cc:3060] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.691327 1614 tablet_service.cc:1505] Processing CreateTablet for tablet c1ca0dfa7e4c4f0fb59fe2bb8fa192ad (DEFAULT_TABLE table=default.Table_1ece0128472c4ef2a4e5f261fe73b3a1 [id=c73b9c050bd94f30bd0a665bed411295]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.674189 1767 tablet_bootstrap.cc:492] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:14.798028 2808 raft_consensus.cc:515] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:14.799000 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505e4fb0755a4caaa462e5f5154bfee2" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:14.800125 2808 leader_election.cc:290] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:14.800344 1447 leader_election.cc:343] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:14.711167 1696 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:14.734879 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ab081e3a2b2445f8b8c73d271416cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.801460 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 529580599bfa4800973b8540ab13d22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.807353 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1ca0dfa7e4c4f0fb59fe2bb8fa192ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.814769 1370 raft_consensus.cc:3060] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.786643 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 520060a9adca4bb0b8616bcc006dce52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.687975 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 9684072368f14cc2942b4e240f90b0f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:14.828058 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6803d23a4714934b85190886611d8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.673085 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 3e09c12bcfd14ab081ab309f9dec4afc (DEFAULT_TABLE table=default.Table_f145f64b26134e0e9fa7c293d5d7842f [id=378d8732b5a24c9899ca74c62d6e88d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.832895 1766 tablet_bootstrap.cc:654] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.837992 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet b1fa439afe5a48a79557019687319af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:14.844151 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0767e8424414437b9dd0a83e1ce6ad4d" } new_table_name: "default.Table_a395ff27a51a4b5e878e524825bdd2cf" modify_external_catalogs: true
I20251024 19:48:14.846335 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9bc761287e4a44ac958794e2ba5b7b79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:14.847168 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bc761287e4a44ac958794e2ba5b7b79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:14.851431 1345 tablet_service.cc:1505] Processing CreateTablet for tablet eee85800b6424ad489cdf7325594b71c (DEFAULT_TABLE table=default.Table_98740f996f06427dbd2383bf1c19bc1b [id=34bd0ecbe5fc43c09fc52ac67637b41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.854079 1482 tablet_service.cc:1505] Processing CreateTablet for tablet eee85800b6424ad489cdf7325594b71c (DEFAULT_TABLE table=default.Table_98740f996f06427dbd2383bf1c19bc1b [id=34bd0ecbe5fc43c09fc52ac67637b41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:14.854602 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee85800b6424ad489cdf7325594b71c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:14.864771 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57e1e21beee44847b1ebeb746f02c372 consensus state has no leader
I20251024 19:48:14.856805 1618 tablet_service.cc:1505] Processing CreateTablet for tablet eee85800b6424ad489cdf7325594b71c (DEFAULT_TABLE table=default.Table_98740f996f06427dbd2383bf1c19bc1b [id=34bd0ecbe5fc43c09fc52ac67637b41a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:14.854450 1586 leader_election.cc:343] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:14.842720 1767 ts_tablet_manager.cc:1403] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.759s user 0.001s sys 0.013s
I20251024 19:48:14.870267 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505e4fb0755a4caaa462e5f5154bfee2" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:14.870545 1643 raft_consensus.cc:3060] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.872285 1776 tablet_replica.cc:333] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:14.878499 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e09c12bcfd14ab081ab309f9dec4afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.886556 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee85800b6424ad489cdf7325594b71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.891810 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0dd268e924c14ae1995b078b85afca08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:14.892490 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dd268e924c14ae1995b078b85afca08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:14.893283 1643 raft_consensus.cc:2468] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:14.898026 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 605205170c994296a5ef0dc568788517. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.899180 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8942872216024d1e94a99643ed4974ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.900534 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaafc2aff3244c8e8aa7bbe4aa59cb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.901384 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee85800b6424ad489cdf7325594b71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.901944 1444 leader_election.cc:304] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:14.902400 2885 raft_consensus.cc:2804] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:14.902527 2885 raft_consensus.cc:697] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:14.902801 2885 consensus_queue.cc:237] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:14.905510 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 529580599bfa4800973b8540ab13d22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.907522 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ab749795a64339abeaf1ce5d6570ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.908201 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce9ad86d489a4df1a7b687514c4d1512. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.910044 1586 leader_election.cc:304] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:14.910121 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505e4fb0755a4caaa462e5f5154bfee2" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:14.911476 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505e4fb0755a4caaa462e5f5154bfee2" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:14.912878 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d933103af272429c95f8d3fc54e427da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.914295 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet b88a6a5237d146f5b6f082ca3af25b2b with delete_type TABLET_DATA_DELETED (replaced by tablet 639836c656f44a819036f4d00e055bd9) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:14.916196 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1ca0dfa7e4c4f0fb59fe2bb8fa192ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.916887 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e09c12bcfd14ab081ab309f9dec4afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:14.917824 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d38154d3ffd34f4b8d7305b1c23a4fae" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:14.920682 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6803d23a4714934b85190886611d8d1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:14.924654 1768 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.133s sys 0.169s
W20251024 19:48:14.924788 1768 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.501s user 0.133s sys 0.169s
I20251024 19:48:14.925670 1768 tablet_bootstrap.cc:492] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:14.925837 1768 ts_tablet_manager.cc:1403] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.509s user 0.135s sys 0.170s
I20251024 19:48:14.927325 1768 raft_consensus.cc:359] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.927547 1768 raft_consensus.cc:385] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.927645 1768 raft_consensus.cc:740] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.697707 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b32126d04e480f8db7b2b20aba5873" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:14.927939 1768 consensus_queue.cc:260] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.928465 1768 ts_tablet_manager.cc:1434] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251024 19:48:14.928750 1768 tablet_bootstrap.cc:492] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:14.929167 1369 raft_consensus.cc:3060] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.930650 1768 tablet_bootstrap.cc:654] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.934285 1776 raft_consensus.cc:2243] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:14.947943 1775 ts_tablet_manager.cc:1929] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:14.948097 1775 log.cc:1199] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/913534ab88dd459e9cac2faebb5492a1
I20251024 19:48:14.948465 1775 ts_tablet_manager.cc:1950] T 913534ab88dd459e9cac2faebb5492a1 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:14.948822 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 685) took 3669 ms. Trace:
I20251024 19:48:14.948911 1775 rpcz_store.cc:276] 1024 19:48:11.279728 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.280463 (+ 735us) service_pool.cc:225] Handling call
1024 19:48:14.672348 (+3391885us) ts_tablet_manager.cc:1151] Deleting tablet 913534ab88dd459e9cac2faebb5492a1
1024 19:48:14.672363 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:14.937261 (+264898us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.947815 (+ 10554us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.948805 (+ 990us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3391505}
I20251024 19:48:14.949404 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 913534ab88dd459e9cac2faebb5492a1 (table default.Table_a83dd919e9ab42179bff96aac9a09cec [id=acce7ccf5ddb40efab47c44305393922]) successfully deleted
I20251024 19:48:14.958352 1768 tablet_bootstrap.cc:492] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:14.958613 1768 ts_tablet_manager.cc:1403] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20251024 19:48:14.959386 1768 raft_consensus.cc:359] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.959627 1768 raft_consensus.cc:385] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.959715 1768 raft_consensus.cc:740] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.959996 1768 consensus_queue.cc:260] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.960610 1768 ts_tablet_manager.cc:1434] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:14.960918 1768 tablet_bootstrap.cc:492] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:14.962343 1768 tablet_bootstrap.cc:654] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.962675 1372 raft_consensus.cc:3060] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:14.970274 2858 raft_consensus.cc:493] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:14.970465 2858 raft_consensus.cc:515] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.971161 2858 leader_election.cc:290] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:14.972368 1365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1fa439afe5a48a79557019687319af6" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:14.972885 1447 leader_election.cc:343] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:14.982102 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 701c93a600964c20a275f5942a2ff863 with delete_type TABLET_DATA_DELETED (replaced by tablet 7da1601a87524ef1a8b5069ba6823843) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:14.983167 2858 raft_consensus.cc:493] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:14.983335 2858 raft_consensus.cc:515] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.984030 2858 leader_election.cc:290] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:14.984809 1365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c512ec26e3874cd7869216b1fb90fa54" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:14.985394 1447 leader_election.cc:343] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:14.986326 1768 tablet_bootstrap.cc:492] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:14.986516 1768 ts_tablet_manager.cc:1403] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.001s
I20251024 19:48:14.987367 1768 raft_consensus.cc:359] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.987624 1768 raft_consensus.cc:385] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:14.987735 1768 raft_consensus.cc:740] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:14.988044 1768 consensus_queue.cc:260] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:14.989135 1768 ts_tablet_manager.cc:1434] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251024 19:48:14.989480 1768 tablet_bootstrap.cc:492] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:14.991353 1768 tablet_bootstrap.cc:654] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:14.983579 1629 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54246 (request call id 785) took 1461 ms. Trace:
I20251024 19:48:14.992583 1629 rpcz_store.cc:276] 1024 19:48:13.522026 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.522069 (+ 43us) service_pool.cc:225] Handling call
1024 19:48:13.532125 (+ 10056us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1024 19:48:13.532148 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1024 19:48:13.771640 (+239492us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:14.983373 (+1211733us) tablet_replica.cc:206] Creating consensus instance
1024 19:48:14.983555 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77442}
I20251024 19:48:14.998467 1305 leader_election.cc:304] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:14.998880 2864 raft_consensus.cc:2749] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:48:14.999214 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1fa439afe5a48a79557019687319af6" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:14.999864 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_aa1ad4ed1e214bb890edff3ebe855022" } new_table_name: "default.Table_43839c3a85d347eca63289bc83306fdc" modify_external_catalogs: true
I20251024 19:48:15.011706 1364 raft_consensus.cc:3060] T 03cdbd1b62b741249ad78562a7edfd22 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:15.018244 2885 raft_consensus.cc:493] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:15.018532 1768 tablet_bootstrap.cc:492] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:15.018500 2885 raft_consensus.cc:515] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.018816 1768 ts_tablet_manager.cc:1403] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
I20251024 19:48:15.019661 1768 raft_consensus.cc:359] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.019905 1768 raft_consensus.cc:385] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:15.020005 1768 raft_consensus.cc:740] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:15.020308 1768 consensus_queue.cc:260] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.021039 1768 ts_tablet_manager.cc:1434] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:15.021442 1768 tablet_bootstrap.cc:492] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:15.021739 2885 leader_election.cc:290] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:15.026268 1768 tablet_bootstrap.cc:654] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:15.026644 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16bc737104ab46b3b071a0760dbc847d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:15.032939 1447 leader_election.cc:343] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.035282 1766 tablet_bootstrap.cc:492] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:15.035516 1766 ts_tablet_manager.cc:1403] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.246s user 0.009s sys 0.001s
I20251024 19:48:15.036736 1766 raft_consensus.cc:359] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.037278 1766 raft_consensus.cc:385] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251024 19:48:15.041147 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af5414e8275845f58717f19fa1b900ed consensus state has no leader
I20251024 19:48:15.041852 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_16ba8561c0c349f0937e04f2ab1ed21a" } modify_external_catalogs: true
I20251024 19:48:15.044452 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cd17bd14f2964c739da271f669e63cd7" } modify_external_catalogs: true
I20251024 19:48:15.044636 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cd17bd14f2964c739da271f669e63cd7" } modify_external_catalogs: true
I20251024 19:48:15.046205 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_867f1b66094349fa8d81af1e89bebb06" } modify_external_catalogs: true
I20251024 19:48:15.047266 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_867f1b66094349fa8d81af1e89bebb06" } modify_external_catalogs: true
I20251024 19:48:15.048441 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_16ba8561c0c349f0937e04f2ab1ed21a" } modify_external_catalogs: true
I20251024 19:48:15.054282 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 4243f11bf2b34b30b10766c19e2c6409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:15.054718 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e9accb7540aa4f2ebe096fd57e7e5105" } modify_external_catalogs: true
I20251024 19:48:15.054980 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e9accb7540aa4f2ebe096fd57e7e5105" } modify_external_catalogs: true
I20251024 19:48:15.056479 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 16bc737104ab46b3b071a0760dbc847d with delete_type TABLET_DATA_DELETED (replaced by tablet 385e71e672154e399f14321eac0d8625) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:15.057260 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 3887fc21726846279fecb02672b9bc07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:15.060457 1183 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 635a132735704cfda69bf7c8deaea81d (table default.Table_933a6697624149369ab3ec28462b466a [id=93a61862707b4a7ba8a0ff349704dc2b]). Expected version 1 got 0
I20251024 19:48:15.060750 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet cef80e4d14bb4e719bceebf43041d079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:15.061144 1183 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 03cdbd1b62b741249ad78562a7edfd22 (table default.Table_42c3a4a03db14d0684721ce6af6960c2 [id=ca742ae585e0406abf43193a23aa9731]). Expected version 2 got 0
I20251024 19:48:15.061432 1183 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 4ffd446c269448d38cc8d987a53aecf9 (table default.Table_bedd8edc24014953ac8c7dcecf18f128 [id=2ae1c808837e48df97c4efb01a684da9]). Expected version 1 got 0
I20251024 19:48:15.065954 1776 raft_consensus.cc:2272] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:15.072021 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f04fa9e7d054461894b1ded30ab195a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.072880 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f04fa9e7d054461894b1ded30ab195a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:15.075067 1444 leader_election.cc:343] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.075206 1444 leader_election.cc:304] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:15.075587 2808 raft_consensus.cc:493] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:15.075709 2808 raft_consensus.cc:515] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.076284 2808 leader_election.cc:290] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:15.076411 2808 raft_consensus.cc:2749] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:15.078538 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68ca2dd438d94bf0b5f1fc69f0a94517 consensus state has no leader
I20251024 19:48:15.081562 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "187bc968c2b049028dd25dfb0264c0aa" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:15.082105 1447 leader_election.cc:343] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.082885 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_86bb3105657540d49403f9b2791e1134"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.083503 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86bb3105657540d49403f9b2791e1134 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.089329 1183 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 4ffd446c269448d38cc8d987a53aecf9 (table default.Table_bedd8edc24014953ac8c7dcecf18f128 [id=2ae1c808837e48df97c4efb01a684da9]). Expected version 1 got 0
I20251024 19:48:15.100589 1364 raft_consensus.cc:3060] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:15.101974 2912 tablet.cc:1722] T 03cdbd1b62b741249ad78562a7edfd22 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:15.066844 2826 raft_consensus.cc:2804] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:15.103668 2826 raft_consensus.cc:697] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:15.104007 2826 consensus_queue.cc:237] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.104094 2885 consensus_queue.cc:1048] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:15.106153 1444 consensus_peers.cc:597] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:15.090438 1767 raft_consensus.cc:359] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.106312 1767 raft_consensus.cc:385] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:15.106370 1767 raft_consensus.cc:740] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:15.106585 1767 consensus_queue.cc:260] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.107180 1767 ts_tablet_manager.cc:1434] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.238s user 0.000s sys 0.002s
I20251024 19:48:15.107510 1767 tablet_bootstrap.cc:492] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:15.092168 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16bc737104ab46b3b071a0760dbc847d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:15.110093 1767 tablet_bootstrap.cc:654] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:15.092763 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "187bc968c2b049028dd25dfb0264c0aa" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:15.111276 1649 raft_consensus.cc:2468] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:15.077045 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c512ec26e3874cd7869216b1fb90fa54" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:15.067236 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 157217us
I20251024 19:48:15.112133 1586 leader_election.cc:304] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
W20251024 19:48:15.112543 1444 leader_election.cc:343] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.112672 1444 leader_election.cc:304] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:15.066746 1776 ts_tablet_manager.cc:1916] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:15.113056 1444 leader_election.cc:304] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:15.113325 2857 raft_consensus.cc:2749] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:15.113426 1444 leader_election.cc:343] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.113489 2857 raft_consensus.cc:2804] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:15.113526 1444 leader_election.cc:304] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:15.113551 2857 raft_consensus.cc:493] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:15.113641 2857 raft_consensus.cc:3060] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:15.113868 2808 raft_consensus.cc:2749] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:15.116432 2857 raft_consensus.cc:515] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.116971 2857 leader_election.cc:290] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:15.117431 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "187bc968c2b049028dd25dfb0264c0aa" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:15.118002 1447 leader_election.cc:343] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.118247 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "187bc968c2b049028dd25dfb0264c0aa" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:15.118445 1643 raft_consensus.cc:3060] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:15.121986 1643 raft_consensus.cc:2468] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:15.122361 1444 leader_election.cc:304] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:15.122794 2885 raft_consensus.cc:2804] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:15.123306 2885 raft_consensus.cc:697] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:15.123543 2828 raft_consensus.cc:2804] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:15.123617 2828 raft_consensus.cc:493] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:15.123678 2828 raft_consensus.cc:3060] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:15.123608 2885 consensus_queue.cc:237] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.126960 2828 raft_consensus.cc:515] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.127717 2828 leader_election.cc:290] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:15.098083 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_94d237360e434747ba2dcdf09d4367cc" } new_table_name: "default.Table_17f78ac6efb148f896b6fc2a2abad27a" modify_external_catalogs: true
I20251024 19:48:15.132529 1183 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 03cdbd1b62b741249ad78562a7edfd22 (table default.Table_42c3a4a03db14d0684721ce6af6960c2 [id=ca742ae585e0406abf43193a23aa9731]). Expected version 2 got 0
I20251024 19:48:15.134949 2921 tablet.cc:1722] T 03cdbd1b62b741249ad78562a7edfd22 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:15.139038 1767 tablet_bootstrap.cc:492] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:15.139559 1767 ts_tablet_manager.cc:1403] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20251024 19:48:15.140561 1767 raft_consensus.cc:359] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.141150 1767 raft_consensus.cc:385] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:15.141255 1767 raft_consensus.cc:740] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:15.141597 1767 consensus_queue.cc:260] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.155954 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 687) took 3875 ms. Trace:
I20251024 19:48:15.156061 1775 rpcz_store.cc:276] 1024 19:48:11.280006 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.281152 (+ 1146us) service_pool.cc:225] Handling call
1024 19:48:15.044168 (+3763016us) ts_tablet_manager.cc:1151] Deleting tablet 46c1e2cef6cf4772b624060afb6fb117
1024 19:48:15.155815 (+111647us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.155931 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3762811}
I20251024 19:48:15.156781 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 688) took 3801 ms. Trace:
I20251024 19:48:15.156863 1775 rpcz_store.cc:276] 1024 19:48:11.355184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.355751 (+ 567us) service_pool.cc:225] Handling call
1024 19:48:15.156659 (+3800908us) ts_tablet_manager.cc:1151] Deleting tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9
1024 19:48:15.156671 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.156764 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3800407}
I20251024 19:48:15.157150 1775 tablet_replica.cc:333] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:15.157336 1775 raft_consensus.cc:2243] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:15.157711 1775 raft_consensus.cc:2272] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:15.158283 1775 ts_tablet_manager.cc:1916] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:15.161396 2921 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:46462 (request call id 800) took 1224 ms. Trace:
I20251024 19:48:15.161612 2921 rpcz_store.cc:276] 1024 19:48:13.936712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.936786 (+ 74us) service_pool.cc:225] Handling call
1024 19:48:15.161377 (+1224591us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1024 19:48:13.937281 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1024 19:48:13.937378 (+ 97us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1024 19:48:13.937396 (+ 18us) alter_schema_op.cc:66] Acquired schema lock
1024 19:48:13.937402 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1024 19:48:13.937449 (+ 47us) alter_schema_op.cc:128] START. Timestamp: 1761335293937385
1024 19:48:13.937453 (+ 4us) op_driver.cc:348] REPLICATION: starting
1024 19:48:13.937827 (+ 374us) log.cc:844] Serialized 173 byte log entry
1024 19:48:15.134517 (+1196690us) op_driver.cc:464] REPLICATION: finished
1024 19:48:15.134920 (+ 403us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1024 19:48:15.141680 (+ 6760us) log.cc:844] Serialized 5 byte log entry
1024 19:48:15.158275 (+ 16595us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.158546 (+ 271us) log.cc:844] Serialized 14 byte log entry
1024 19:48:15.161327 (+ 2781us) alter_schema_op.cc:73] Released schema lock
1024 19:48:15.161333 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":347,"prepare.queue_time_us":68,"prepare.run_cpu_time_us":628,"prepare.run_wall_time_us":653,"raft.queue_time_us":71,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":186,"replication_time_us":1197019,"thread_start_us":641,"threads_started":1}]]}
I20251024 19:48:15.138758 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 95f62010a24f432c84c89345f9b02b2f
I20251024 19:48:15.139010 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9f791083b4964ff9b0f5fc575478892c
W20251024 19:48:15.156558 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 46c1e2cef6cf4772b624060afb6fb117 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46c1e2cef6cf4772b624060afb6fb117 already in progress: creating tablet
W20251024 19:48:15.166831 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 already in progress: creating tablet
I20251024 19:48:15.167577 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f791083b4964ff9b0f5fc575478892c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:15.168002 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 95f62010a24f432c84c89345f9b02b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:15.168002 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f791083b4964ff9b0f5fc575478892c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:15.169912 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 95f62010a24f432c84c89345f9b02b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:15.170287 1627 tablet_service.cc:1552] Processing DeleteTablet for tablet 9f791083b4964ff9b0f5fc575478892c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:15.176808 1642 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.000s sys 0.008s
W20251024 19:48:15.176932 1642 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.518s user 0.000s sys 0.008s
I20251024 19:48:15.181656 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 95f62010a24f432c84c89345f9b02b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:15.187206 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d3ce6e24c37a4f35999df5fe4dc17266"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.187835 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3ce6e24c37a4f35999df5fe4dc17266 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.190196 1191 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 03cdbd1b62b741249ad78562a7edfd22 (table default.Table_42c3a4a03db14d0684721ce6af6960c2 [id=ca742ae585e0406abf43193a23aa9731]). Expected version 2 got 0
I20251024 19:48:15.191380 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fdf3a5f81257407d8b5d28c7f7cab1b7" } modify_external_catalogs: true
I20251024 19:48:15.191437 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3e8db9d6df6542088fef2a3b687daddd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:15.191589 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fdf3a5f81257407d8b5d28c7f7cab1b7" } modify_external_catalogs: true
W20251024 19:48:15.191951 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e8db9d6df6542088fef2a3b687daddd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.193233 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_50d055bf7fc34e8fa68f1a188aea6218" } new_table_name: "default.Table_256d5a49b85c4a07b9c5e1839ee9bee6" modify_external_catalogs: true
W20251024 19:48:15.194669 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22fa56d227f54eaf9f58ca116e730931 consensus state has no leader
I20251024 19:48:15.198544 1774 ts_tablet_manager.cc:1929] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:15.198777 1774 log.cc:1199] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/ad81d80697984ff1834a7d795e0f4b9b
I20251024 19:48:15.199081 2925 tablet.cc:1722] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251024 19:48:15.199203 1774 ts_tablet_manager.cc:1950] T ad81d80697984ff1834a7d795e0f4b9b P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:15.199545 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 644) took 3812 ms. Trace:
I20251024 19:48:15.199644 1774 rpcz_store.cc:276] 1024 19:48:11.386830 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.387094 (+ 264us) service_pool.cc:225] Handling call
1024 19:48:14.589967 (+3202873us) ts_tablet_manager.cc:1151] Deleting tablet ad81d80697984ff1834a7d795e0f4b9b
1024 19:48:14.589992 (+ 25us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.010304 (+420312us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.198418 (+188114us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.199524 (+ 1106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3202745}
I20251024 19:48:15.200062 1774 tablet_replica.cc:333] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:15.200232 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet ad81d80697984ff1834a7d795e0f4b9b (table default.Table_61b7f6c428aa4bd9a65c1d66171c59c9 [id=6c604c6f86f944a285866376d9a5ce65]) successfully deleted
I20251024 19:48:15.200294 1774 raft_consensus.cc:2243] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:15.200474 1774 raft_consensus.cc:2272] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:15.201076 1774 ts_tablet_manager.cc:1916] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:15.037644 1766 raft_consensus.cc:740] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:15.204243 1775 ts_tablet_manager.cc:1929] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:15.216717 1775 log.cc:1199] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/b79801594e95434cbb8fe7b75b14602e
I20251024 19:48:15.217125 1775 ts_tablet_manager.cc:1950] T b79801594e95434cbb8fe7b75b14602e P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:15.217478 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 690) took 3838 ms. Trace:
I20251024 19:48:15.217599 2927 tablet.cc:1722] T 037c8cc1e53447109fafc047c02f196f P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251024 19:48:15.217557 1775 rpcz_store.cc:276] 1024 19:48:11.378554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.378603 (+ 49us) service_pool.cc:225] Handling call
1024 19:48:15.157110 (+3778507us) ts_tablet_manager.cc:1151] Deleting tablet b79801594e95434cbb8fe7b75b14602e
1024 19:48:15.157118 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.201025 (+ 43907us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.204149 (+ 3124us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.217459 (+ 13310us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3778304}
I20251024 19:48:15.217927 1775 tablet_replica.cc:333] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:15.218115 1775 raft_consensus.cc:2243] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:15.218194 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet b79801594e95434cbb8fe7b75b14602e (table default.Table_aa7c283553a841e9ab2fa21fb6485a11 [id=7ec9391c52b1424eb52850f2c6ba4779]) successfully deleted
I20251024 19:48:15.218281 1775 raft_consensus.cc:2272] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:15.218803 1775 ts_tablet_manager.cc:1916] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:15.212221 1766 consensus_queue.cc:260] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.222214 1766 ts_tablet_manager.cc:1434] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.187s user 0.001s sys 0.001s
I20251024 19:48:15.222508 1766 tablet_bootstrap.cc:492] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:15.223932 1766 tablet_bootstrap.cc:654] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:15.228698 2919 tablet.cc:1722] T 03cdbd1b62b741249ad78562a7edfd22 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:15.229871 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet bc61fded84b24e00b299e5d342076708 from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:15.234994 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 103663us
I20251024 19:48:15.236284 1766 tablet_bootstrap.cc:492] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:15.237362 1766 ts_tablet_manager.cc:1403] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20251024 19:48:15.238782 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e29abfa3bc8944aea7d1acd03e3eb897" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:15.239138 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e29abfa3bc8944aea7d1acd03e3eb897" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:15.239362 1510 raft_consensus.cc:3060] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:15.239568 1767 ts_tablet_manager.cc:1434] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.100s user 0.003s sys 0.002s
I20251024 19:48:15.239849 1767 tablet_bootstrap.cc:492] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:15.240331 2561 raft_consensus.cc:493] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:15.240442 2561 raft_consensus.cc:515] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.240991 2561 leader_election.cc:290] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:15.241458 1767 tablet_bootstrap.cc:654] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:15.242311 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ec77ed043045411b85ba06840af39907"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:15.242599 2921 tablet.cc:1722] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251024 19:48:15.242969 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec77ed043045411b85ba06840af39907 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.243636 1510 raft_consensus.cc:2468] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:15.245369 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be8126459ef541f39bc133f27c7d0f59" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:15.245366 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be8126459ef541f39bc133f27c7d0f59" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:15.245909 1444 leader_election.cc:343] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:15.246419 1447 leader_election.cc:343] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.246524 1447 leader_election.cc:304] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:15.247270 2857 raft_consensus.cc:2749] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:15.248337 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e2e3a2eac58e42fa8d80c405e3ec136a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.248972 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2e3a2eac58e42fa8d80c405e3ec136a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.251696 1768 tablet_bootstrap.cc:492] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:15.253998 1766 raft_consensus.cc:359] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.254189 1766 raft_consensus.cc:385] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:15.254246 1766 raft_consensus.cc:740] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:15.254782 1776 ts_tablet_manager.cc:1929] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.4
I20251024 19:48:15.254900 1776 log.cc:1199] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/452f77091adc4cf09de2231f2f6d845b
I20251024 19:48:15.255157 1768 ts_tablet_manager.cc:1403] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.234s user 0.013s sys 0.028s
I20251024 19:48:15.255313 1776 ts_tablet_manager.cc:1950] T 452f77091adc4cf09de2231f2f6d845b P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:15.255658 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 730) took 3509 ms. Trace:
I20251024 19:48:15.255731 1766 consensus_queue.cc:260] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.255743 1776 rpcz_store.cc:276] 1024 19:48:11.746546 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.747148 (+ 602us) service_pool.cc:225] Handling call
1024 19:48:14.853365 (+3106217us) ts_tablet_manager.cc:1151] Deleting tablet 452f77091adc4cf09de2231f2f6d845b
1024 19:48:14.858706 (+ 5341us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.162533 (+303827us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.254659 (+ 92126us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.255641 (+ 982us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61978,"tablet-delete.queue_time_us":3090571}
I20251024 19:48:15.256203 1776 tablet_replica.cc:333] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:15.256325 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 452f77091adc4cf09de2231f2f6d845b (table default.Table_5bfbf5d7f5764b96a95bf3ac0e0aba69 [id=9a49120c5b654676a63cd3966eaf31a9]) successfully deleted
I20251024 19:48:15.256458 1776 raft_consensus.cc:2243] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:15.256649 1776 raft_consensus.cc:2272] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:15.257068 1768 raft_consensus.cc:359] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.259630 2921 tablet.cc:1722] T 037c8cc1e53447109fafc047c02f196f P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251024 19:48:15.268595 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87575045817948d9b4980944b2e62614 consensus state has no leader
I20251024 19:48:15.269948 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 8ba85164015448d4a688a473cce90c5c
I20251024 19:48:15.272881 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ba85164015448d4a688a473cce90c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:15.275950 2561 consensus_queue.cc:1048] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:15.279284 1776 ts_tablet_manager.cc:1916] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:15.258909 1586 leader_election.cc:343] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.276827 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ba85164015448d4a688a473cce90c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:15.280232 1586 leader_election.cc:304] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
W20251024 19:48:15.280980 1586 consensus_peers.cc:597] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:15.281378 1586 leader_election.cc:343] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.282073 2828 raft_consensus.cc:2749] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:15.284600 1586 leader_election.cc:343] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.287473 1768 raft_consensus.cc:385] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:15.288005 1768 raft_consensus.cc:740] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:15.288360 2925 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:54246 (request call id 803) took 1650 ms. Trace:
I20251024 19:48:15.288671 1768 consensus_queue.cc:260] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.273173 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 8ba85164015448d4a688a473cce90c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:15.281469 2864 raft_consensus.cc:493] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:15.291169 2864 raft_consensus.cc:515] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.291915 2864 leader_election.cc:290] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:15.293601 2927 tablet.cc:1722] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251024 19:48:15.293761 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07e75c7f35544bec82cb994dea5ee4ef" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:15.288951 2925 rpcz_store.cc:276] 1024 19:48:13.637434 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.718220 (+ 80786us) service_pool.cc:225] Handling call
1024 19:48:15.288343 (+1570123us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1024 19:48:13.763682 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1024 19:48:13.776135 (+ 12453us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1024 19:48:13.776143 (+ 8us) alter_schema_op.cc:66] Acquired schema lock
1024 19:48:13.776150 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1024 19:48:13.776189 (+ 39us) alter_schema_op.cc:128] START. Timestamp: 1761335293776037
1024 19:48:13.776192 (+ 3us) op_driver.cc:348] REPLICATION: starting
1024 19:48:13.776342 (+ 150us) log.cc:844] Serialized 173 byte log entry
1024 19:48:15.130989 (+1354647us) op_driver.cc:464] REPLICATION: finished
1024 19:48:15.199048 (+ 68059us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1024 19:48:15.201119 (+ 2071us) log.cc:844] Serialized 5 byte log entry
1024 19:48:15.282653 (+ 81534us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.282905 (+ 252us) log.cc:844] Serialized 14 byte log entry
1024 19:48:15.288295 (+ 5390us) alter_schema_op.cc:73] Released schema lock
1024 19:48:15.288302 (+ 7us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":67974,"prepare.queue_time_us":645,"prepare.run_cpu_time_us":20488,"prepare.run_wall_time_us":934858,"replication_time_us":1354749,"thread_start_us":1006761,"threads_started":2,"wal-append.queue_time_us":617096}]]}
I20251024 19:48:15.294234 2912 tablet.cc:1722] T 037c8cc1e53447109fafc047c02f196f P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251024 19:48:15.296777 1767 tablet_bootstrap.cc:492] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:15.296963 1767 ts_tablet_manager.cc:1403] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.002s
I20251024 19:48:15.297765 1767 raft_consensus.cc:359] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.298022 1767 raft_consensus.cc:385] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:15.298107 1767 raft_consensus.cc:740] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:15.298339 1767 consensus_queue.cc:260] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.299707 1767 ts_tablet_manager.cc:1434] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251024 19:48:15.300488 1767 tablet_bootstrap.cc:492] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:15.300596 2912 tablet.cc:1722] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251024 19:48:15.307345 1586 leader_election.cc:304] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:15.309096 2826 raft_consensus.cc:2804] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:15.309382 2826 raft_consensus.cc:697] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:15.309736 2826 consensus_queue.cc:237] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.314074 1767 tablet_bootstrap.cc:654] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:15.316339 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07e75c7f35544bec82cb994dea5ee4ef" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:15.316623 1644 raft_consensus.cc:2393] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
I20251024 19:48:15.317117 1305 leader_election.cc:304] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:15.317643 2864 raft_consensus.cc:3060] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:15.321012 2864 raft_consensus.cc:2749] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:15.322916 1766 ts_tablet_manager.cc:1434] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.085s user 0.001s sys 0.001s
I20251024 19:48:15.325793 1766 tablet_bootstrap.cc:492] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:15.327375 1766 tablet_bootstrap.cc:654] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:15.336001 1775 ts_tablet_manager.cc:1929] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:15.336133 1775 log.cc:1199] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/dc14b0cdb7f342ac956a7857ab3f35d7
I20251024 19:48:15.336529 1775 ts_tablet_manager.cc:1950] T dc14b0cdb7f342ac956a7857ab3f35d7 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:15.336858 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 691) took 3955 ms. Trace:
I20251024 19:48:15.336951 1775 rpcz_store.cc:276] 1024 19:48:11.381318 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.384505 (+ 3187us) service_pool.cc:225] Handling call
1024 19:48:15.217876 (+3833371us) ts_tablet_manager.cc:1151] Deleting tablet dc14b0cdb7f342ac956a7857ab3f35d7
1024 19:48:15.217887 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.323946 (+106059us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.335890 (+ 11944us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.336840 (+ 950us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3833174}
I20251024 19:48:15.337463 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet dc14b0cdb7f342ac956a7857ab3f35d7 (table default.Table_be61dd09c4c94f568d723c4bace3a038 [id=66be3fc672ea4a9e9fb5dde3ae5e5b62]) successfully deleted
W20251024 19:48:15.340798 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ef2add387da44868dbd2b5724ee7d86 consensus state has no leader
I20251024 19:48:15.342267 1183 catalog_manager.cc:5649] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:15.342723 1183 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet e03c488d0f264684b0897a81a04d7466 (table default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8]). Expected version 2 got 0
I20251024 19:48:15.343015 1183 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 03cdbd1b62b741249ad78562a7edfd22 (table default.Table_42c3a4a03db14d0684721ce6af6960c2 [id=ca742ae585e0406abf43193a23aa9731]). Expected version 2 got 0
I20251024 19:48:15.343396 1183 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 187925082cdd4923a6e9148bf9f270fd (table default.Table_55f8c66bd97343438051a98af12d8c2c [id=33f5ac73298543d0b5af77a27e897b79]). Expected version 1 got 0
I20251024 19:48:15.343729 1183 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 635a132735704cfda69bf7c8deaea81d (table default.Table_933a6697624149369ab3ec28462b466a [id=93a61862707b4a7ba8a0ff349704dc2b]). Expected version 1 got 0
I20251024 19:48:15.347214 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 4606b064ce4245b6a07e157324dcb70f (DEFAULT_TABLE table=default.Table_7fd523f97f3e4c549d353c15d2ac6e36 [id=41f9cb82d93b45d0a3f6cab02df1c051]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.347489 2828 consensus_queue.cc:1048] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:15.347687 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4606b064ce4245b6a07e157324dcb70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.349838 1483 tablet_service.cc:1505] Processing CreateTablet for tablet a27237314b684588aba9934065d9337b (DEFAULT_TABLE table=default.Table_fe7a480771e74942ab56f7fdd69ecf23 [id=abdd8431661f4d53a82b563ccff4aee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.350158 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27237314b684588aba9934065d9337b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.350425 2921 tablet.cc:1722] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251024 19:48:15.351663 2937 tablet.cc:1722] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251024 19:48:15.353811 1488 tablet_service.cc:1505] Processing CreateTablet for tablet ebac8a8f215d4fc3800a666f0e1397ce (DEFAULT_TABLE table=default.Table_b6ce782f9e28425585cf22bc3c1bc98b [id=011747e2742d42549f83008560004060]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.354108 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebac8a8f215d4fc3800a666f0e1397ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.358116 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 68d85226ac5b4e4ab22a49dc419e375a (DEFAULT_TABLE table=default.Table_177f4c4e8d3b437aa14bb9e8a383510a [id=2c91e008871b48918546301a0eaf4c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.358453 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68d85226ac5b4e4ab22a49dc419e375a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.361646 2921 tablet.cc:1722] T 635a132735704cfda69bf7c8deaea81d P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:15.363178 1490 tablet_service.cc:1505] Processing CreateTablet for tablet db93abf6776b4c80af128f37499d9d2c (DEFAULT_TABLE table=default.Table_e5d0d22a76d5435c8861460346530bd4 [id=8454b8f14a6e4ca8af8b34df2a02e7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.363257 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 9b7ab1bf5b9549afb150802a209527eb (DEFAULT_TABLE table=default.Table_1ff2aa8257594c2fa59a715522aea99d [id=af54747e41a24ddc9db73df69e383170]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.363382 2561 raft_consensus.cc:493] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:15.363533 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db93abf6776b4c80af128f37499d9d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.363489 2561 raft_consensus.cc:515] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.363531 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 4606b064ce4245b6a07e157324dcb70f (DEFAULT_TABLE table=default.Table_7fd523f97f3e4c549d353c15d2ac6e36 [id=41f9cb82d93b45d0a3f6cab02df1c051]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.363797 1349 tablet_service.cc:1505] Processing CreateTablet for tablet ebac8a8f215d4fc3800a666f0e1397ce (DEFAULT_TABLE table=default.Table_b6ce782f9e28425585cf22bc3c1bc98b [id=011747e2742d42549f83008560004060]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.363265 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 81e3e2f8082d4c229e617f34f5eb5a78 (DEFAULT_TABLE table=default.Table_d25c3d97a9a54883942fe92efe382f58 [id=a55f5edd1e1c4f069fd8109606003b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.364061 2561 leader_election.cc:290] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:15.364428 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 81e3e2f8082d4c229e617f34f5eb5a78 (DEFAULT_TABLE table=default.Table_d25c3d97a9a54883942fe92efe382f58 [id=a55f5edd1e1c4f069fd8109606003b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.364563 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e3e2f8082d4c229e617f34f5eb5a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.364754 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b7ab1bf5b9549afb150802a209527eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.364898 1340 tablet_service.cc:1505] Processing CreateTablet for tablet db93abf6776b4c80af128f37499d9d2c (DEFAULT_TABLE table=default.Table_e5d0d22a76d5435c8861460346530bd4 [id=8454b8f14a6e4ca8af8b34df2a02e7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.365433 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 68d85226ac5b4e4ab22a49dc419e375a (DEFAULT_TABLE table=default.Table_177f4c4e8d3b437aa14bb9e8a383510a [id=2c91e008871b48918546301a0eaf4c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.365839 1338 tablet_service.cc:1505] Processing CreateTablet for tablet a27237314b684588aba9934065d9337b (DEFAULT_TABLE table=default.Table_fe7a480771e74942ab56f7fdd69ecf23 [id=abdd8431661f4d53a82b563ccff4aee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.366298 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 9b7ab1bf5b9549afb150802a209527eb (DEFAULT_TABLE table=default.Table_1ff2aa8257594c2fa59a715522aea99d [id=af54747e41a24ddc9db73df69e383170]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.366734 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "758df87acbdb4adaa83e1ecbc44e7cf1" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:15.366744 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 2d5a37e494924d6286de266656692637 (DEFAULT_TABLE table=default.Table_80521123c9bb431dae3bf90e358d8276 [id=925278ecea774a95b51f0188ea83c5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.366912 1366 raft_consensus.cc:2468] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:15.367807 1447 leader_election.cc:304] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:15.368165 2561 raft_consensus.cc:2804] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251024 19:48:15.368257 2561 raft_consensus.cc:493] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:15.368312 2561 raft_consensus.cc:3060] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Advancing to term 2
I20251024 19:48:15.372027 2561 raft_consensus.cc:515] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.374145 2561 leader_election.cc:290] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:15.375391 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "758df87acbdb4adaa83e1ecbc44e7cf1" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:15.375583 1366 raft_consensus.cc:3060] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Advancing to term 2
I20251024 19:48:15.376405 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 2d5a37e494924d6286de266656692637 (DEFAULT_TABLE table=default.Table_80521123c9bb431dae3bf90e358d8276 [id=925278ecea774a95b51f0188ea83c5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.376988 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d5a37e494924d6286de266656692637. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.381019 1418 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:15.382050 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 703441e1af8e4356afd22ef68dac87a2 (DEFAULT_TABLE table=default.Table_af70ad0ef8864f3a83c8376e84e7ddd3 [id=d0e3d80b2b7643d4a48fc48e4dd4b446]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.384187 1366 raft_consensus.cc:2468] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 2.
I20251024 19:48:15.385252 1447 leader_election.cc:304] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:15.385520 2858 raft_consensus.cc:493] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:15.385648 2858 raft_consensus.cc:515] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.386193 2858 leader_election.cc:290] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:15.386322 2858 raft_consensus.cc:2804] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Leader election won for term 2
I20251024 19:48:15.386428 2858 raft_consensus.cc:697] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 2 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:15.386417 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 68d85226ac5b4e4ab22a49dc419e375a (DEFAULT_TABLE table=default.Table_177f4c4e8d3b437aa14bb9e8a383510a [id=2c91e008871b48918546301a0eaf4c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.386730 2858 consensus_queue.cc:237] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.386888 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68d85226ac5b4e4ab22a49dc419e375a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.389249 1341 tablet_service.cc:1505] Processing CreateTablet for tablet bb6b253a7c73485391366147a9bbc7f9 (DEFAULT_TABLE table=default.Table_28cc3637c8e04cd291b968fe9631ae13 [id=42d3e90fc6514ca9bf5cc3cb8e7b1632]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.390630 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 4606b064ce4245b6a07e157324dcb70f (DEFAULT_TABLE table=default.Table_7fd523f97f3e4c549d353c15d2ac6e36 [id=41f9cb82d93b45d0a3f6cab02df1c051]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.390447 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3887fc21726846279fecb02672b9bc07" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:15.391038 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4606b064ce4245b6a07e157324dcb70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.391111 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 703441e1af8e4356afd22ef68dac87a2 (DEFAULT_TABLE table=default.Table_af70ad0ef8864f3a83c8376e84e7ddd3 [id=d0e3d80b2b7643d4a48fc48e4dd4b446]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.392082 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703441e1af8e4356afd22ef68dac87a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:15.392118 1444 leader_election.cc:343] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 758df87acbdb4adaa83e1ecbc44e7cf1
W20251024 19:48:15.392498 1444 leader_election.cc:343] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 758df87acbdb4adaa83e1ecbc44e7cf1
W20251024 19:48:15.393477 1447 leader_election.cc:343] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.395503 2919 tablet.cc:1722] T 635a132735704cfda69bf7c8deaea81d P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:15.397452 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 81e3e2f8082d4c229e617f34f5eb5a78 (DEFAULT_TABLE table=default.Table_d25c3d97a9a54883942fe92efe382f58 [id=a55f5edd1e1c4f069fd8109606003b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.398319 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e3e2f8082d4c229e617f34f5eb5a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.397838 1627 tablet_service.cc:1505] Processing CreateTablet for tablet a27237314b684588aba9934065d9337b (DEFAULT_TABLE table=default.Table_fe7a480771e74942ab56f7fdd69ecf23 [id=abdd8431661f4d53a82b563ccff4aee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.399120 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27237314b684588aba9934065d9337b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.402058 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 2d5a37e494924d6286de266656692637 (DEFAULT_TABLE table=default.Table_80521123c9bb431dae3bf90e358d8276 [id=925278ecea774a95b51f0188ea83c5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.402386 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d5a37e494924d6286de266656692637. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.405149 1491 tablet_service.cc:1505] Processing CreateTablet for tablet bb6b253a7c73485391366147a9bbc7f9 (DEFAULT_TABLE table=default.Table_28cc3637c8e04cd291b968fe9631ae13 [id=42d3e90fc6514ca9bf5cc3cb8e7b1632]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.405490 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb6b253a7c73485391366147a9bbc7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.405783 1624 tablet_service.cc:1505] Processing CreateTablet for tablet bb6b253a7c73485391366147a9bbc7f9 (DEFAULT_TABLE table=default.Table_28cc3637c8e04cd291b968fe9631ae13 [id=42d3e90fc6514ca9bf5cc3cb8e7b1632]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.406075 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb6b253a7c73485391366147a9bbc7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.409657 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 91bb076b4b0d4b04955043248460d04c (DEFAULT_TABLE table=default.Table_e7975a91f4674c4fa7904e3b7bc780dc [id=85d27c777170436191cefa0d378e91c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.409804 1625 tablet_service.cc:1505] Processing CreateTablet for tablet ebac8a8f215d4fc3800a666f0e1397ce (DEFAULT_TABLE table=default.Table_b6ce782f9e28425585cf22bc3c1bc98b [id=011747e2742d42549f83008560004060]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.410123 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebac8a8f215d4fc3800a666f0e1397ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.414086 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 9b7ab1bf5b9549afb150802a209527eb (DEFAULT_TABLE table=default.Table_1ff2aa8257594c2fa59a715522aea99d [id=af54747e41a24ddc9db73df69e383170]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.414399 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b7ab1bf5b9549afb150802a209527eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.415138 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 703441e1af8e4356afd22ef68dac87a2 (DEFAULT_TABLE table=default.Table_af70ad0ef8864f3a83c8376e84e7ddd3 [id=d0e3d80b2b7643d4a48fc48e4dd4b446]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.415495 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703441e1af8e4356afd22ef68dac87a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.417148 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 91bb076b4b0d4b04955043248460d04c (DEFAULT_TABLE table=default.Table_e7975a91f4674c4fa7904e3b7bc780dc [id=85d27c777170436191cefa0d378e91c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.417538 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91bb076b4b0d4b04955043248460d04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.418898 1616 tablet_service.cc:1505] Processing CreateTablet for tablet db93abf6776b4c80af128f37499d9d2c (DEFAULT_TABLE table=default.Table_e5d0d22a76d5435c8861460346530bd4 [id=8454b8f14a6e4ca8af8b34df2a02e7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.419201 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db93abf6776b4c80af128f37499d9d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.419128 2808 consensus_queue.cc:1048] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:15.418448 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3887fc21726846279fecb02672b9bc07" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:15.420897 1333 tablet_service.cc:1505] Processing CreateTablet for tablet fe9cb79e470948e5b0af65b6b668913a (DEFAULT_TABLE table=default.Table_1f888bb5a760477c88e82d58b11aa364 [id=99a7cdb786364e708e183c7514979029]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:15.424330 1444 leader_election.cc:343] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.424474 1444 leader_election.cc:304] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:15.424639 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 91bb076b4b0d4b04955043248460d04c (DEFAULT_TABLE table=default.Table_e7975a91f4674c4fa7904e3b7bc780dc [id=85d27c777170436191cefa0d378e91c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.425036 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91bb076b4b0d4b04955043248460d04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.425557 2561 raft_consensus.cc:2749] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:15.430373 1486 tablet_service.cc:1505] Processing CreateTablet for tablet fe9cb79e470948e5b0af65b6b668913a (DEFAULT_TABLE table=default.Table_1f888bb5a760477c88e82d58b11aa364 [id=99a7cdb786364e708e183c7514979029]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.430794 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe9cb79e470948e5b0af65b6b668913a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.432093 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4606b064ce4245b6a07e157324dcb70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.432277 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b7ab1bf5b9549afb150802a209527eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.432459 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e3e2f8082d4c229e617f34f5eb5a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.432616 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe9cb79e470948e5b0af65b6b668913a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.432817 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebac8a8f215d4fc3800a666f0e1397ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.437700 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d5a37e494924d6286de266656692637. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.437920 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91bb076b4b0d4b04955043248460d04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.438123 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27237314b684588aba9934065d9337b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.438391 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb6b253a7c73485391366147a9bbc7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.443756 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68d85226ac5b4e4ab22a49dc419e375a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.444391 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db93abf6776b4c80af128f37499d9d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.444869 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703441e1af8e4356afd22ef68dac87a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:15.446621 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 96748.8us
I20251024 19:48:15.448621 2937 tablet.cc:1722] T 635a132735704cfda69bf7c8deaea81d P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:15.451948 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 4f10340e106b473eb715b74b129835b5 (DEFAULT_TABLE table=default.Table_6b8a2c3a14404cab96880c1a8630f56c [id=28922f53a4d14fe2b8fdb29c01a5dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.452644 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f10340e106b473eb715b74b129835b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.416877 2920 raft_consensus.cc:493] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251024 19:48:15.454531 1444 consensus_peers.cc:597] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:15.454782 1629 tablet_service.cc:1505] Processing CreateTablet for tablet fe9cb79e470948e5b0af65b6b668913a (DEFAULT_TABLE table=default.Table_1f888bb5a760477c88e82d58b11aa364 [id=99a7cdb786364e708e183c7514979029]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.455199 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe9cb79e470948e5b0af65b6b668913a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.460387 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 4f10340e106b473eb715b74b129835b5 (DEFAULT_TABLE table=default.Table_6b8a2c3a14404cab96880c1a8630f56c [id=28922f53a4d14fe2b8fdb29c01a5dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.461126 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f10340e106b473eb715b74b129835b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.463126 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 4f10340e106b473eb715b74b129835b5 (DEFAULT_TABLE table=default.Table_6b8a2c3a14404cab96880c1a8630f56c [id=28922f53a4d14fe2b8fdb29c01a5dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.463673 1490 tablet_service.cc:1505] Processing CreateTablet for tablet b072e9b0eb4046a083f04436a24db018 (DEFAULT_TABLE table=default.Table_8840cc4de84c4e1c95015f5440515ed7 [id=6754c5962958496690f9dbe4d628fb0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.464051 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b072e9b0eb4046a083f04436a24db018. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.469473 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4865fd997563401387e8f190fb26e86a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:15.470202 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f10340e106b473eb715b74b129835b5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:15.470392 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4865fd997563401387e8f190fb26e86a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.472182 1617 tablet_service.cc:1505] Processing CreateTablet for tablet b072e9b0eb4046a083f04436a24db018 (DEFAULT_TABLE table=default.Table_8840cc4de84c4e1c95015f5440515ed7 [id=6754c5962958496690f9dbe4d628fb0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.472393 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_09c35871ae6345f8bd692a8e416323e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:15.472618 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b072e9b0eb4046a083f04436a24db018. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:15.473037 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09c35871ae6345f8bd692a8e416323e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.475837 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fdc631c49a3d4e6b89b84a4af697b305" } new_table_name: "default.Table_727cead5e96a4e04af23383cb5503560" modify_external_catalogs: true
I20251024 19:48:15.478592 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0e854ff9b0b84dcbbdea3ec2c97a7aef" } new_table_name: "default.Table_7711dc5051cb4ec08eb9367dd8698a33" modify_external_catalogs: true
I20251024 19:48:15.478865 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9cbca7d5292d44c49d2d550aa9e18e92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.479524 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cbca7d5292d44c49d2d550aa9e18e92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.482198 2925 tablet.cc:1717] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97: Skipping requested alter to schema version 3, tablet already version 3
I20251024 19:48:15.482736 1483 tablet_service.cc:1505] Processing CreateTablet for tablet f16ef86c5eb24489a66d6cd4bfa804fa (DEFAULT_TABLE table=default.Table_4b8dc796eac74a329a4e65bbf0410b3b [id=6d161dc7a5b54771a8b14b1f0cbb0291]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.483134 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f16ef86c5eb24489a66d6cd4bfa804fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.484376 2925 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:54246 (request call id 832) took 1057 ms. Trace:
I20251024 19:48:15.464308 2920 raft_consensus.cc:515] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.487668 2923 raft_consensus.cc:493] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:15.488338 1615 tablet_service.cc:1505] Processing CreateTablet for tablet f16ef86c5eb24489a66d6cd4bfa804fa (DEFAULT_TABLE table=default.Table_4b8dc796eac74a329a4e65bbf0410b3b [id=6d161dc7a5b54771a8b14b1f0cbb0291]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.488596 1774 ts_tablet_manager.cc:1929] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:15.488705 1774 log.cc:1199] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/7352d84226db467dad31ae5941842b80
I20251024 19:48:15.488803 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f16ef86c5eb24489a66d6cd4bfa804fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.489029 1774 ts_tablet_manager.cc:1950] T 7352d84226db467dad31ae5941842b80 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:15.489348 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 645) took 4031 ms. Trace:
I20251024 19:48:15.489420 1774 rpcz_store.cc:276] 1024 19:48:11.457669 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.457719 (+ 50us) service_pool.cc:225] Handling call
1024 19:48:15.200009 (+3742290us) ts_tablet_manager.cc:1151] Deleting tablet 7352d84226db467dad31ae5941842b80
1024 19:48:15.200020 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.312626 (+112606us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.488476 (+175850us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.489332 (+ 856us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3742084}
I20251024 19:48:15.489868 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 655) took 3927 ms. Trace:
I20251024 19:48:15.489949 1774 rpcz_store.cc:276] 1024 19:48:11.561906 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.561966 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:15.489721 (+3927755us) ts_tablet_manager.cc:1151] Deleting tablet 82cf75ee148e45b29da3343b3b068781
1024 19:48:15.489732 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.489851 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3927490}
I20251024 19:48:15.490275 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 657) took 3925 ms. Trace:
I20251024 19:48:15.491031 2920 leader_election.cc:290] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:15.491214 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 7352d84226db467dad31ae5941842b80 (table default.Table_71bcb15d38b842efbdfc0fb90171e71e [id=d115a3523a5746918a343221a41c9589]) successfully deleted
W20251024 19:48:15.491885 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 82cf75ee148e45b29da3343b3b068781 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82cf75ee148e45b29da3343b3b068781 already in progress: creating tablet
I20251024 19:48:15.487828 2923 raft_consensus.cc:515] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.484479 2925 rpcz_store.cc:276] 1024 19:48:14.427251 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.558437 (+131186us) service_pool.cc:225] Handling call
1024 19:48:15.484358 (+925921us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1024 19:48:14.698644 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1024 19:48:14.698852 (+ 208us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1024 19:48:15.289184 (+590332us) alter_schema_op.cc:66] Acquired schema lock
1024 19:48:15.289211 (+ 27us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1024 19:48:15.289250 (+ 39us) alter_schema_op.cc:128] START. Timestamp: 1761335295289218
1024 19:48:15.289255 (+ 5us) op_driver.cc:348] REPLICATION: starting
1024 19:48:15.289440 (+ 185us) log.cc:844] Serialized 173 byte log entry
1024 19:48:15.482060 (+192620us) op_driver.cc:464] REPLICATION: finished
1024 19:48:15.482159 (+ 99us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1024 19:48:15.484273 (+ 2114us) log.cc:844] Serialized 94 byte log entry
1024 19:48:15.484314 (+ 41us) alter_schema_op.cc:73] Released schema lock
1024 19:48:15.484319 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":36,"prepare.queue_time_us":139809,"prepare.run_cpu_time_us":12479,"prepare.run_wall_time_us":590879,"replication_time_us":192758}]]}
I20251024 19:48:15.501613 2923 leader_election.cc:290] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:15.511570 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 58134.2us
I20251024 19:48:15.514120 2921 tablet.cc:1717] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e: Skipping requested alter to schema version 3, tablet already version 3
I20251024 19:48:15.518559 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 584cbe0c396841f794e665aae3ebbd91
I20251024 19:48:15.519443 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c9d965a31a7f4ba3bed33813b02d7dfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:15.520179 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f518918f7bfe409fae8f42ac3d059872" } modify_external_catalogs: true
I20251024 19:48:15.520375 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f518918f7bfe409fae8f42ac3d059872" } modify_external_catalogs: true
I20251024 19:48:15.520635 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2cbeaac19e834ecf98e5b8afc952b9da
I20251024 19:48:15.520855 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 584cbe0c396841f794e665aae3ebbd91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:15.521294 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ce9762cc2ae44e2dbf2404da0554444c" } modify_external_catalogs: true
I20251024 19:48:15.521477 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ce9762cc2ae44e2dbf2404da0554444c" } modify_external_catalogs: true
W20251024 19:48:15.523765 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9d965a31a7f4ba3bed33813b02d7dfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.524562 2885 consensus_queue.cc:1048] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:15.524626 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6ab1b43d9b5e4537bd38fdb50f100fc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.525132 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ab1b43d9b5e4537bd38fdb50f100fc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.525832 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cbeaac19e834ecf98e5b8afc952b9da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:15.532241 2561 consensus_queue.cc:1048] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:15.541087 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f with delete_type TABLET_DATA_DELETED (replaced by tablet 9a1031d785b646b095ece9be666d2a77) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:15.542886 1619 tablet_service.cc:1552] Processing DeleteTablet for tablet 584cbe0c396841f794e665aae3ebbd91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:15.544042 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2461bf28f0bd44ac8e86c5bd843a6718"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.544668 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2461bf28f0bd44ac8e86c5bd843a6718 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.548033 1619 tablet_service.cc:1552] Processing DeleteTablet for tablet 452f77091adc4cf09de2231f2f6d845b with delete_type TABLET_DATA_DELETED (replaced by tablet f709db684b6c49cbaeef3bf8d76f5828) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:15.558316 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cbeaac19e834ecf98e5b8afc952b9da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:15.570021 2940 log.cc:927] Time spent T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20251024 19:48:15.493255 2864 raft_consensus.cc:493] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:15.572211 2864 raft_consensus.cc:515] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.572925 2864 leader_election.cc:290] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:15.573266 1366 raft_consensus.cc:3060] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:15.572113 1774 rpcz_store.cc:276] 1024 19:48:11.565106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.565155 (+ 49us) service_pool.cc:225] Handling call
1024 19:48:15.490175 (+3925020us) ts_tablet_manager.cc:1151] Deleting tablet 8876b7b38e744212a3722dbe5bcc5596
1024 19:48:15.490183 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.490262 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3924860}
I20251024 19:48:15.573964 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 659) took 3998 ms. Trace:
W20251024 19:48:15.574302 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 8876b7b38e744212a3722dbe5bcc5596 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8876b7b38e744212a3722dbe5bcc5596 already in progress: creating tablet
I20251024 19:48:15.575455 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ffd446c269448d38cc8d987a53aecf9" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:15.495237 1344 tablet_service.cc:1505] Processing CreateTablet for tablet f16ef86c5eb24489a66d6cd4bfa804fa (DEFAULT_TABLE table=default.Table_4b8dc796eac74a329a4e65bbf0410b3b [id=6d161dc7a5b54771a8b14b1f0cbb0291]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.514400 1348 tablet_service.cc:1505] Processing CreateTablet for tablet b072e9b0eb4046a083f04436a24db018 (DEFAULT_TABLE table=default.Table_8840cc4de84c4e1c95015f5440515ed7 [id=6754c5962958496690f9dbe4d628fb0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.525652 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cbeaac19e834ecf98e5b8afc952b9da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:15.576812 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b072e9b0eb4046a083f04436a24db018. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.577461 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ffd446c269448d38cc8d987a53aecf9" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:15.577720 1644 raft_consensus.cc:2393] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
I20251024 19:48:15.578153 1305 leader_election.cc:304] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:15.531682 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 584cbe0c396841f794e665aae3ebbd91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:15.578655 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f16ef86c5eb24489a66d6cd4bfa804fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.581657 1768 ts_tablet_manager.cc:1434] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.326s user 0.019s sys 0.015s
I20251024 19:48:15.582037 1768 tablet_bootstrap.cc:492] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:15.583525 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce593bda7c8547bb9cf2b30ca6820776 consensus state has no leader
I20251024 19:48:15.583961 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d3ce6e24c37a4f35999df5fe4dc17266" } new_table_name: "default.Table_8da6aee6329246318cec0d90763e6dde" modify_external_catalogs: true
I20251024 19:48:15.585690 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 63518b583ac94fa6abde420e176ec8ba
I20251024 19:48:15.586661 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ab448d5388b84b3489200f3a17c24edb" } modify_external_catalogs: true
I20251024 19:48:15.586899 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ab448d5388b84b3489200f3a17c24edb" } modify_external_catalogs: true
I20251024 19:48:15.589607 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 63518b583ac94fa6abde420e176ec8ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:15.590066 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f00719c20934655bc744446194f7401 consensus state has no leader
I20251024 19:48:15.590626 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a9ab11ad10c74cd1be885016fa225113"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.591228 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9ab11ad10c74cd1be885016fa225113 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.593173 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 63518b583ac94fa6abde420e176ec8ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:15.596822 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b809079c0a9a4540814e848853c374cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.597550 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b809079c0a9a4540814e848853c374cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.599905 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 63518b583ac94fa6abde420e176ec8ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:15.606166 1192 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_42c3a4a03db14d0684721ce6af6960c2 [id=ca742ae585e0406abf43193a23aa9731] alter complete (version 2)
I20251024 19:48:15.607780 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet b79801594e95434cbb8fe7b75b14602e with delete_type TABLET_DATA_DELETED (replaced by tablet 3e6f8764e9ae472b9ac35a032182aed1) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:15.612026 1775 tablet_replica.cc:333] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:15.612246 1775 raft_consensus.cc:2243] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:15.612534 1767 tablet_bootstrap.cc:492] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:15.612618 1775 raft_consensus.cc:2272] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:15.612687 1767 ts_tablet_manager.cc:1403] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.312s user 0.007s sys 0.003s
I20251024 19:48:15.613198 1775 ts_tablet_manager.cc:1916] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:15.614058 1767 raft_consensus.cc:359] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.614285 1767 raft_consensus.cc:385] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:15.614334 1767 raft_consensus.cc:740] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:15.618606 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_22b072e068244a37b59bbac02c3dfb75" } new_table_name: "default.Table_8b808e9cdcbc4edd90f1804171f93ac4" modify_external_catalogs: true
I20251024 19:48:15.620942 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a478a67f294e4fdcba8d2f2b421d26fd" } new_table_name: "default.Table_e427007b23f447b99ee7db28f8f4c085" modify_external_catalogs: true
I20251024 19:48:15.625291 1766 tablet_bootstrap.cc:492] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:15.625438 1766 ts_tablet_manager.cc:1403] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.302s user 0.001s sys 0.001s
I20251024 19:48:15.626298 1766 raft_consensus.cc:359] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.626479 1766 raft_consensus.cc:385] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:15.626545 1766 raft_consensus.cc:740] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:15.626766 1766 consensus_queue.cc:260] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.627324 1766 ts_tablet_manager.cc:1434] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:15.627666 1766 tablet_bootstrap.cc:492] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:15.632649 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c512ec26e3874cd7869216b1fb90fa54" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:15.632838 1505 raft_consensus.cc:2468] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:15.642262 1586 leader_election.cc:304] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:15.647285 2569 raft_consensus.cc:2804] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:15.647429 2569 raft_consensus.cc:493] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:15.647506 2569 raft_consensus.cc:3060] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:15.649176 1187 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet e03c488d0f264684b0897a81a04d7466 (table default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8]). Expected version 2 got 0
I20251024 19:48:15.652652 2569 raft_consensus.cc:515] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.653623 2569 leader_election.cc:290] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:15.653986 1586 leader_election.cc:343] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: f51c56337ebb450aaf62dcc81ef63ae8
I20251024 19:48:15.655323 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c512ec26e3874cd7869216b1fb90fa54" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:15.655578 1505 raft_consensus.cc:3060] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:15.651441 1177 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 4f00719c20934655bc744446194f7401 (table default.Table_727cead5e96a4e04af23383cb5503560 [id=f62531b6d396440e8f44ae943c4db403], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4f00719c20934655bc744446194f7401 consensus state has no leader
I20251024 19:48:15.664409 2919 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:54246 (request call id 825) took 1551 ms. Trace:
I20251024 19:48:15.664578 2919 rpcz_store.cc:276] 1024 19:48:14.112580 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.112634 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:15.664376 (+1551742us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1024 19:48:14.458235 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1024 19:48:14.458433 (+ 198us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1024 19:48:14.458444 (+ 11us) alter_schema_op.cc:66] Acquired schema lock
1024 19:48:14.458452 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1024 19:48:14.458489 (+ 37us) alter_schema_op.cc:128] START. Timestamp: 1761335294458431
1024 19:48:14.458492 (+ 3us) op_driver.cc:348] REPLICATION: starting
1024 19:48:14.458712 (+ 220us) log.cc:844] Serialized 173 byte log entry
1024 19:48:15.392532 (+933820us) op_driver.cc:464] REPLICATION: finished
1024 19:48:15.395476 (+ 2944us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1024 19:48:15.396521 (+ 1045us) log.cc:844] Serialized 5 byte log entry
1024 19:48:15.630931 (+234410us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.631241 (+ 310us) log.cc:844] Serialized 14 byte log entry
1024 19:48:15.664303 (+ 33062us) alter_schema_op.cc:73] Released schema lock
1024 19:48:15.664313 (+ 10us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2744,"prepare.queue_time_us":296626,"prepare.run_cpu_time_us":7914,"prepare.run_wall_time_us":200936,"replication_time_us":887903,"thread_start_us":585208,"threads_started":2,"wal-append.queue_time_us":199111}]]}
I20251024 19:48:15.669854 1767 consensus_queue.cc:260] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.672679 1767 ts_tablet_manager.cc:1434] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.060s user 0.000s sys 0.003s
I20251024 19:48:15.684311 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3219a7b4a2474607bd3df6f6d3e35642"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.685019 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3219a7b4a2474607bd3df6f6d3e35642 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.686228 1767 tablet_bootstrap.cc:492] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:48:15.686538 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 776a235a8055419e975008f6c607ca57 has no consensus state
I20251024 19:48:15.686972 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_809ec8b3a0fc4fa08457269a52eae680"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.687531 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_809ec8b3a0fc4fa08457269a52eae680 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.688649 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet bf241a9566c34b7b911b69aaf4de06a2
I20251024 19:48:15.574034 1774 rpcz_store.cc:276] 1024 19:48:11.575944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.576010 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:15.573839 (+3997829us) ts_tablet_manager.cc:1151] Deleting tablet cb0405765c6146cbbb187af96a60be2e
1024 19:48:15.573850 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.573948 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3997644}
I20251024 19:48:15.690555 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 661) took 4022 ms. Trace:
I20251024 19:48:15.690636 1774 rpcz_store.cc:276] 1024 19:48:11.668496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.668574 (+ 78us) service_pool.cc:225] Handling call
1024 19:48:15.690413 (+4021839us) ts_tablet_manager.cc:1151] Deleting tablet cbd321c059e34834a6257fab5e6d14bb
1024 19:48:15.690426 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.690533 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4021552}
I20251024 19:48:15.690986 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 663) took 4020 ms. Trace:
I20251024 19:48:15.691056 1774 rpcz_store.cc:276] 1024 19:48:11.670767 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.670825 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:15.690885 (+4020060us) ts_tablet_manager.cc:1151] Deleting tablet 50cea450eda9485daba40bf875a1b90c
1024 19:48:15.690893 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.690970 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4019835}
I20251024 19:48:15.692523 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet bf241a9566c34b7b911b69aaf4de06a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:15.692885 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 605205170c994296a5ef0dc568788517
I20251024 19:48:15.694092 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet 605205170c994296a5ef0dc568788517 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:15.694295 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet bf241a9566c34b7b911b69aaf4de06a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:15.693635 1776 ts_tablet_manager.cc:1929] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:15.701457 2569 raft_consensus.cc:493] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:15.701946 2569 raft_consensus.cc:515] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.702929 2569 leader_election.cc:290] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:15.711277 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 605205170c994296a5ef0dc568788517 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:15.713297 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f83b9290f00145a780318a43249eec12" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:15.713500 1508 raft_consensus.cc:2468] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:15.715142 1505 raft_consensus.cc:2468] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:15.717028 1766 tablet_bootstrap.cc:654] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:15.718869 2569 raft_consensus.cc:493] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:15.719067 2569 raft_consensus.cc:515] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:15.719307 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4846799df5b242ce8f19c034320c8a1d consensus state has no leader
I20251024 19:48:15.719892 2569 leader_election.cc:290] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:15.721959 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e72407cfebf14ad2be5ee53e418fda62" } modify_external_catalogs: true
I20251024 19:48:15.722174 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e72407cfebf14ad2be5ee53e418fda62" } modify_external_catalogs: true
I20251024 19:48:15.723778 2808 consensus_queue.cc:1048] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:15.724948 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9e14814a2db64e0b9602e41f926f3b62" } modify_external_catalogs: true
I20251024 19:48:15.725142 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9e14814a2db64e0b9602e41f926f3b62" } modify_external_catalogs: true
W20251024 19:48:15.728197 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 424bce2f496b4a4fa74e57794f6a11eb consensus state has no leader
I20251024 19:48:15.729521 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6c5115e46f6746acb0d4fa01a9227c6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.730160 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c5115e46f6746acb0d4fa01a9227c6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.731221 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1fa439afe5a48a79557019687319af6" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:15.731416 1505 raft_consensus.cc:2468] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:15.732322 1586 leader_election.cc:304] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:15.732659 1586 leader_election.cc:304] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:15.743010 2923 raft_consensus.cc:2804] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:15.743145 2923 raft_consensus.cc:493] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:15.743218 2923 raft_consensus.cc:3060] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:15.747107 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b8a9d4b3797a4bd6af7122890b92c30e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:15.747349 2923 raft_consensus.cc:515] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:15.747818 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8a9d4b3797a4bd6af7122890b92c30e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.748126 2923 leader_election.cc:290] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:15.748306 2923 raft_consensus.cc:2804] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:15.748445 2923 raft_consensus.cc:697] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:15.748584 1766 tablet_bootstrap.cc:492] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:15.748739 1766 ts_tablet_manager.cc:1403] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.121s user 0.004s sys 0.000s
I20251024 19:48:15.748773 2923 consensus_queue.cc:237] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.749742 1766 raft_consensus.cc:359] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.749917 1766 raft_consensus.cc:385] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:15.749974 1766 raft_consensus.cc:740] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:15.750222 1766 consensus_queue.cc:260] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.750784 1766 ts_tablet_manager.cc:1434] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:15.751150 1766 tablet_bootstrap.cc:492] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:15.751953 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 662) took 4081 ms. Trace:
I20251024 19:48:15.752046 1774 rpcz_store.cc:276] 1024 19:48:11.670632 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.671065 (+ 433us) service_pool.cc:225] Handling call
1024 19:48:15.691366 (+4020301us) ts_tablet_manager.cc:1151] Deleting tablet 6f7b6d15f25d4c2a905da6597ba8d98a
1024 19:48:15.691376 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.751928 (+ 60552us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4020020}
W20251024 19:48:15.752632 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet cb0405765c6146cbbb187af96a60be2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb0405765c6146cbbb187af96a60be2e already in progress: creating tablet
W20251024 19:48:15.752967 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet cbd321c059e34834a6257fab5e6d14bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbd321c059e34834a6257fab5e6d14bb already in progress: creating tablet
W20251024 19:48:15.753288 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 50cea450eda9485daba40bf875a1b90c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50cea450eda9485daba40bf875a1b90c already in progress: creating tablet
I20251024 19:48:15.754586 1586 leader_election.cc:304] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:15.755295 2569 raft_consensus.cc:2804] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:15.755384 2569 raft_consensus.cc:493] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:15.755385 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f83b9290f00145a780318a43249eec12" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:15.755471 2569 raft_consensus.cc:3060] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:15.755545 1505 raft_consensus.cc:3060] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:15.755685 1774 tablet_replica.cc:333] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:15.758193 1505 raft_consensus.cc:2468] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:15.758514 1766 tablet_bootstrap.cc:654] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:15.760979 2569 raft_consensus.cc:515] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.761257 1775 ts_tablet_manager.cc:1929] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:15.761453 1775 log.cc:1199] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/7352d84226db467dad31ae5941842b80
I20251024 19:48:15.761659 2569 leader_election.cc:290] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:15.762018 1775 ts_tablet_manager.cc:1950] T 7352d84226db467dad31ae5941842b80 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:15.762358 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1fa439afe5a48a79557019687319af6" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:15.762524 1505 raft_consensus.cc:3060] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:15.762692 1586 leader_election.cc:304] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:15.763782 2864 raft_consensus.cc:493] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af3952de050140d383f6b0de7666fa97)
I20251024 19:48:15.763835 2569 raft_consensus.cc:2804] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:15.763958 2569 raft_consensus.cc:697] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:15.763921 2864 raft_consensus.cc:515] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.764199 2569 consensus_queue.cc:237] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.765754 1505 raft_consensus.cc:2468] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:15.766222 1586 leader_election.cc:304] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:15.766462 2926 raft_consensus.cc:2804] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:15.766557 1774 raft_consensus.cc:2243] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:15.766592 2926 raft_consensus.cc:697] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:15.766829 2926 consensus_queue.cc:237] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.752274 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet bf241a9566c34b7b911b69aaf4de06a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:15.696591 1767 tablet_bootstrap.cc:654] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:15.770881 1768 tablet_bootstrap.cc:654] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:15.774531 2864 leader_election.cc:290] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:15.775309 2936 raft_consensus.cc:2749] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:15.778772 1774 raft_consensus.cc:2272] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:15.779323 1774 ts_tablet_manager.cc:1916] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:15.782456 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 689) took 4404 ms. Trace:
I20251024 19:48:15.782559 1775 rpcz_store.cc:276] 1024 19:48:11.378430 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.387259 (+ 8829us) service_pool.cc:225] Handling call
1024 19:48:15.611975 (+4224716us) ts_tablet_manager.cc:1151] Deleting tablet 7352d84226db467dad31ae5941842b80
1024 19:48:15.611984 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.727684 (+115700us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.761075 (+ 33391us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.782437 (+ 21362us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4224554}
I20251024 19:48:15.783073 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 7352d84226db467dad31ae5941842b80 (table default.Table_71bcb15d38b842efbdfc0fb90171e71e [id=d115a3523a5746918a343221a41c9589]) successfully deleted
I20251024 19:48:15.784042 1775 tablet_replica.cc:333] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:15.784514 1775 raft_consensus.cc:2243] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:15.785329 1775 raft_consensus.cc:2272] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:15.787478 1775 ts_tablet_manager.cc:1916] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:15.791169 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b2be2ca906f34c9c855f39abd4548bf0" } modify_external_catalogs: true
I20251024 19:48:15.791368 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b2be2ca906f34c9c855f39abd4548bf0" } modify_external_catalogs: true
I20251024 19:48:15.791904 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_bab51dadce54487d96884e3ef240e684" } new_table_name: "default.Table_85b8c022a7cf44aa8440d324f8b12973" modify_external_catalogs: true
W20251024 19:48:15.794090 1447 consensus_peers.cc:597] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:15.794350 1447 consensus_peers.cc:597] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:15.794674 2885 consensus_queue.cc:1048] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:15.796216 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_cde3670bf0d04dc0b2049193bcee10d4" } new_table_name: "default.Table_3e341daa717746478259c784488beecf" modify_external_catalogs: true
I20251024 19:48:15.799662 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 605205170c994296a5ef0dc568788517 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:15.802167 1767 tablet_bootstrap.cc:492] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:15.802521 1767 ts_tablet_manager.cc:1403] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.116s user 0.002s sys 0.003s
I20251024 19:48:15.805147 1767 raft_consensus.cc:359] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.806725 1767 raft_consensus.cc:385] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:15.807402 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c512ec26e3874cd7869216b1fb90fa54" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:15.807204 1767 raft_consensus.cc:740] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
W20251024 19:48:15.808014 1305 leader_election.cc:343] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: dc14b0cdb7f342ac956a7857ab3f35d7
I20251024 19:48:15.808207 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c512ec26e3874cd7869216b1fb90fa54" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:15.808566 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f51c56337ebb450aaf62dcc81ef63ae8" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:15.808424 1767 consensus_queue.cc:260] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.815526 1767 ts_tablet_manager.cc:1434] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s
I20251024 19:48:15.817068 1767 tablet_bootstrap.cc:492] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:15.830792 1775 ts_tablet_manager.cc:1929] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:15.830952 1775 log.cc:1199] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/9b89a90bb4ca447fbaf34fe39001c261
I20251024 19:48:15.831288 1775 ts_tablet_manager.cc:1950] T 9b89a90bb4ca447fbaf34fe39001c261 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:15.832099 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 693) took 4438 ms. Trace:
I20251024 19:48:15.832185 1775 rpcz_store.cc:276] 1024 19:48:11.393592 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.393643 (+ 51us) service_pool.cc:225] Handling call
1024 19:48:15.783992 (+4390349us) ts_tablet_manager.cc:1151] Deleting tablet 9b89a90bb4ca447fbaf34fe39001c261
1024 19:48:15.784002 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.799735 (+ 15733us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.811691 (+ 11956us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.832078 (+ 20387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4390147}
I20251024 19:48:15.832492 1775 tablet_replica.cc:333] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:15.834533 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 9b89a90bb4ca447fbaf34fe39001c261 (table default.Table_f37eb464f5144d1a8135a5f1a44cee4f [id=2fda7fc5a71646b79f5d848dca570135]) successfully deleted
I20251024 19:48:15.836956 2561 raft_consensus.cc:493] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:15.837109 2561 raft_consensus.cc:515] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.838104 2561 leader_election.cc:290] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:15.842757 2937 tablet.cc:1717] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532: Skipping requested alter to schema version 3, tablet already version 3
I20251024 19:48:15.844830 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4193750331874d7ba9bbe6279c3a8e37" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:15.845204 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f83b9290f00145a780318a43249eec12" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:15.845850 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1fa439afe5a48a79557019687319af6" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:15.846196 1447 leader_election.cc:343] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.846356 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f83b9290f00145a780318a43249eec12" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:15.846910 1775 raft_consensus.cc:2243] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 2 LEADER]: Raft consensus shutting down.
I20251024 19:48:15.846974 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1fa439afe5a48a79557019687319af6" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:15.847788 1775 pending_rounds.cc:70] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e: Trying to abort 1 pending ops.
W20251024 19:48:15.847958 1586 leader_election.cc:343] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:15.848482 1586 leader_election.cc:343] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:15.849167 1308 leader_election.cc:343] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: dc14b0cdb7f342ac956a7857ab3f35d7
I20251024 19:48:15.849313 1308 leader_election.cc:304] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:15.849402 1775 pending_rounds.cc:77] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7214429369906479104 op_type: NO_OP noop_request { }
I20251024 19:48:15.849642 2936 raft_consensus.cc:2749] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:48:15.849828 1775 raft_consensus.cc:2889] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:15.852397 1775 raft_consensus.cc:2272] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:15.853562 1775 ts_tablet_manager.cc:1916] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:15.855221 1767 tablet_bootstrap.cc:654] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:15.856889 1586 leader_election.cc:343] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.857023 1586 leader_election.cc:304] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:15.857510 2926 raft_consensus.cc:2749] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:15.857784 1586 leader_election.cc:343] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:15.858134 1586 leader_election.cc:343] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:15.858393 1586 leader_election.cc:343] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:15.858693 1586 leader_election.cc:343] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.860396 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d488cc5bda8c4725a45d2a2cadb7fab7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.861076 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d488cc5bda8c4725a45d2a2cadb7fab7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.862929 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4193750331874d7ba9bbe6279c3a8e37" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:15.863780 1444 leader_election.cc:343] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:15.863909 1444 leader_election.cc:304] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:15.864360 2561 raft_consensus.cc:2749] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:15.867650 1776 log.cc:1199] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/6a7e76a3cac443ec8d2ab466075aae6b
I20251024 19:48:15.868809 1776 ts_tablet_manager.cc:1950] T 6a7e76a3cac443ec8d2ab466075aae6b P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:15.869195 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 731) took 4087 ms. Trace:
I20251024 19:48:15.869331 1776 rpcz_store.cc:276] 1024 19:48:11.781589 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.784377 (+ 2788us) service_pool.cc:225] Handling call
1024 19:48:15.256147 (+3471770us) ts_tablet_manager.cc:1151] Deleting tablet 6a7e76a3cac443ec8d2ab466075aae6b
1024 19:48:15.256160 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.608517 (+352357us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.684999 (+ 76482us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.869179 (+184180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3471546}
I20251024 19:48:15.870121 1776 tablet_replica.cc:333] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:15.870373 1776 raft_consensus.cc:2243] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:15.870777 1776 raft_consensus.cc:2272] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:15.871647 1776 ts_tablet_manager.cc:1916] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:15.872620 1329 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:54172 (request call id 144) took 1009 ms. Trace:
I20251024 19:48:15.874430 1329 rpcz_store.cc:276] 1024 19:48:14.863247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.863307 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:15.007600 (+144293us) spinlock_profiling.cc:243] Waited 78.5 ms on lock 0x6160001c2154. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:15.046320 (+ 38720us) spinlock_profiling.cc:243] Waited 36.6 ms on lock 0x616000162d54. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:15.321706 (+275386us) spinlock_profiling.cc:243] Waited 94.1 ms on lock 0x616000132530. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f9890e20829 00007f9893944478 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:15.872580 (+550874us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":481240576}
I20251024 19:48:15.878523 1776 ts_tablet_manager.cc:1929] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:15.878731 1776 log.cc:1199] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/27b32126d04e480f8db7b2b20aba5873
I20251024 19:48:15.878739 1331 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:54172 (request call id 142) took 1072 ms. Trace:
I20251024 19:48:15.878845 1331 rpcz_store.cc:276] 1024 19:48:14.805864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.806452 (+ 588us) service_pool.cc:225] Handling call
1024 19:48:15.033357 (+226905us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x6160001b3a54. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:15.362117 (+328760us) spinlock_profiling.cc:243] Waited 64.6 ms on lock 0x616000132530. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f9890e20829 00007f9893944478 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:15.830569 (+468452us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x6160001c6a30. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f9890e20829 00007f9893944478 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:15.878684 (+ 48115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":189765248}
I20251024 19:48:15.879487 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:54172 (request call id 141) took 1073 ms. Trace:
I20251024 19:48:15.879518 1776 ts_tablet_manager.cc:1950] T 27b32126d04e480f8db7b2b20aba5873 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:15.879597 1332 rpcz_store.cc:276] 1024 19:48:14.805768 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.806008 (+ 240us) service_pool.cc:225] Handling call
1024 19:48:15.031903 (+225895us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x6160001b3a54. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:15.060562 (+ 28659us) spinlock_profiling.cc:243] Waited 27.8 ms on lock 0x616000163354. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:15.879439 (+818877us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":100934784}
I20251024 19:48:15.880013 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 732) took 3985 ms. Trace:
I20251024 19:48:15.880129 1776 rpcz_store.cc:276] 1024 19:48:11.894198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.949673 (+ 55475us) service_pool.cc:225] Handling call
1024 19:48:15.870073 (+3920400us) ts_tablet_manager.cc:1151] Deleting tablet 27b32126d04e480f8db7b2b20aba5873
1024 19:48:15.870083 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.875244 (+ 5161us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.878425 (+ 3181us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.879996 (+ 1571us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3920210}
I20251024 19:48:15.881029 1774 ts_tablet_manager.cc:1929] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:15.881158 1774 log.cc:1199] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/6a7e76a3cac443ec8d2ab466075aae6b
I20251024 19:48:15.881560 1774 ts_tablet_manager.cc:1950] T 6a7e76a3cac443ec8d2ab466075aae6b P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:15.881927 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 665) took 4070 ms. Trace:
I20251024 19:48:15.882150 1774 rpcz_store.cc:276] 1024 19:48:11.811872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.811918 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:15.752293 (+3940375us) ts_tablet_manager.cc:1151] Deleting tablet 6a7e76a3cac443ec8d2ab466075aae6b
1024 19:48:15.755632 (+ 3339us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.778638 (+ 23006us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x6160001a0b54. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fac9100 00007f9890e16ae3 00007f98939d3287 00007f9893a062d4 00007f98939e6d70 00007f98939e6afd 0000000000a8867c 00007f9883c49d7f 00007f9883c4d04b 00007f9883c4cec8
1024 19:48:15.828517 (+ 49879us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.880931 (+ 52414us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.881888 (+ 957us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24510848,"tablet-delete.queue_time_us":3940199}
I20251024 19:48:15.882534 1774 tablet_replica.cc:333] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:15.882753 1774 raft_consensus.cc:2243] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:15.882999 1774 raft_consensus.cc:2272] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:15.883617 1774 ts_tablet_manager.cc:1916] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:15.890921 1775 ts_tablet_manager.cc:1929] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251024 19:48:15.891054 1775 log.cc:1199] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/758df87acbdb4adaa83e1ecbc44e7cf1
I20251024 19:48:15.891435 1775 ts_tablet_manager.cc:1950] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:15.891798 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 692) took 4498 ms. Trace:
I20251024 19:48:15.891888 1775 rpcz_store.cc:276] 1024 19:48:11.393462 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.393934 (+ 472us) service_pool.cc:225] Handling call
1024 19:48:15.832444 (+4438510us) ts_tablet_manager.cc:1151] Deleting tablet 758df87acbdb4adaa83e1ecbc44e7cf1
1024 19:48:15.832456 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.875461 (+ 43005us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.890806 (+ 15345us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:15.891780 (+ 974us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13200,"tablet-delete.queue_time_us":4438361}
I20251024 19:48:15.892225 1775 tablet_replica.cc:333] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:15.892414 1775 raft_consensus.cc:2243] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:15.892671 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 758df87acbdb4adaa83e1ecbc44e7cf1 (table default.Table_003cb0c156394f9b8b0ba1b4ac52ce4c [id=b81a7d71911844c1be495ade2ceaf1d4]) successfully deleted
I20251024 19:48:15.892777 1775 raft_consensus.cc:2272] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:15.893338 1775 ts_tablet_manager.cc:1916] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:15.894397 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f1b82a4bcb2d4cca842b1be702afda63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.895149 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1b82a4bcb2d4cca842b1be702afda63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.899276 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 27b32126d04e480f8db7b2b20aba5873 (table default.Table_a372d22e91854206b3523ecd087303ca [id=dfffb9501a0549198d17b197ef05871e]) successfully deleted
I20251024 19:48:15.899748 1776 tablet_replica.cc:333] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:15.900028 1776 raft_consensus.cc:2243] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:15.900277 1776 raft_consensus.cc:2272] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:15.909956 1776 ts_tablet_manager.cc:1916] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:15.911749 1330 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:54172 (request call id 140) took 1553 ms. Trace:
I20251024 19:48:15.911860 1330 rpcz_store.cc:276] 1024 19:48:14.358283 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.358447 (+ 164us) service_pool.cc:225] Handling call
1024 19:48:14.692926 (+334479us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x6160001d2654. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:15.005221 (+312295us) spinlock_profiling.cc:243] Waited 61.2 ms on lock 0x6160001c2154. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:15.042597 (+ 37376us) spinlock_profiling.cc:243] Waited 35 ms on lock 0x616000162d54. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:15.303227 (+260630us) spinlock_profiling.cc:243] Waited 78.8 ms on lock 0x616000132530. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f9890e20829 00007f9893944478 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:15.911714 (+608487us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1165468416}
I20251024 19:48:15.924103 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_039c8e8c0a8946a59f10e06691e5b946"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:15.924799 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_039c8e8c0a8946a59f10e06691e5b946 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:15.932015 1766 tablet_bootstrap.cc:492] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:15.932225 1766 ts_tablet_manager.cc:1403] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.181s user 0.011s sys 0.012s
I20251024 19:48:15.933255 1766 raft_consensus.cc:359] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.933692 1766 raft_consensus.cc:385] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:15.933799 1766 raft_consensus.cc:740] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:15.933880 1768 tablet_bootstrap.cc:492] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:15.934096 1766 consensus_queue.cc:260] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:15.934829 1766 ts_tablet_manager.cc:1434] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:15.937896 1768 ts_tablet_manager.cc:1403] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.356s user 0.004s sys 0.008s
I20251024 19:48:15.938732 1768 raft_consensus.cc:359] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.942458 1766 tablet_bootstrap.cc:492] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:15.948117 1189 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet e03c488d0f264684b0897a81a04d7466 (table default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8]). Expected version 2 got 0
I20251024 19:48:15.948566 1189 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 187925082cdd4923a6e9148bf9f270fd (table default.Table_55f8c66bd97343438051a98af12d8c2c [id=33f5ac73298543d0b5af77a27e897b79]). Expected version 1 got 0
I20251024 19:48:15.948994 1189 catalog_manager.cc:5649] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:15.949445 1189 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet e29abfa3bc8944aea7d1acd03e3eb897 (table default.Table_55339e02660843c3a8536a4b7a06831e [id=ba30503223dd4979824ec9c1a50a8ae2]). Expected version 1 got 0
I20251024 19:48:15.950320 1629 tablet_service.cc:1505] Processing CreateTablet for tablet af918843c74940888c709d9a05aa6e0f (DEFAULT_TABLE table=default.Table_ec77ed043045411b85ba06840af39907 [id=a6c04f5524384b75bfcedd008167716b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.950551 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 563897855a6e4c23803bf4ae34115850 (DEFAULT_TABLE table=default.Table_e2e3a2eac58e42fa8d80c405e3ec136a [id=db4af0d9193f45118561b3bd78834210]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.950320 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 3c708051e9754fe8afc45154b0a4f30c (DEFAULT_TABLE table=default.Table_3e8db9d6df6542088fef2a3b687daddd [id=df27e3eb9ee648aa9825431915fcd8af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.952667 1483 tablet_service.cc:1505] Processing CreateTablet for tablet af918843c74940888c709d9a05aa6e0f (DEFAULT_TABLE table=default.Table_ec77ed043045411b85ba06840af39907 [id=a6c04f5524384b75bfcedd008167716b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.953629 2561 consensus_queue.cc:1048] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:15.954237 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 776a235a8055419e975008f6c607ca57 (DEFAULT_TABLE table=default.Table_8da6aee6329246318cec0d90763e6dde [id=30533d5803cf420197caceaa22294ca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.954695 1614 tablet_service.cc:1505] Processing CreateTablet for tablet f953a1b7d93743a5b1cdcbe9bc164923 (DEFAULT_TABLE table=default.Table_86bb3105657540d49403f9b2791e1134 [id=17daaf93b7ac4f8090683b6929098d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.956306 1625 tablet_service.cc:1505] Processing CreateTablet for tablet b8d30b2fdeec410fa582a05db0968ebb (DEFAULT_TABLE table=default.Table_9bc761287e4a44ac958794e2ba5b7b79 [id=b6210ccdd78245f09ae700c7907b427c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.958606 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 6e419003a8644b0dbd4ce325a0b49d1e (DEFAULT_TABLE table=default.Table_dbfe837064fb41238410f42ada6b39a7 [id=6244042d94f442bf953c891d5690ae78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.959901 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af918843c74940888c709d9a05aa6e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.960078 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776a235a8055419e975008f6c607ca57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.962054 1187 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_81cbf739a62f4e90a669a2638e7cc36f [id=5bc052b3e0f04699b4b391bcec33617d] alter complete (version 3)
I20251024 19:48:15.963491 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f953a1b7d93743a5b1cdcbe9bc164923. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.963886 2923 raft_consensus.cc:493] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:15.964010 2923 raft_consensus.cc:515] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:15.964663 2923 leader_election.cc:290] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:15.965920 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8d30b2fdeec410fa582a05db0968ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.969835 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 3cdd85a3ca7545eea953e149a6c0d572 (DEFAULT_TABLE table=default.Table_0dd268e924c14ae1995b078b85afca08 [id=b1b6eae1d691498c8d5ca1f837ccc288]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.970175 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cdd85a3ca7545eea953e149a6c0d572. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.971829 1629 tablet_service.cc:1505] Processing CreateTablet for tablet ad21fd1f5d1f45fb99a0560e3405d69f (DEFAULT_TABLE table=default.Table_01c58de0bf6845b08ff6fab88827cdec [id=4cdc1ba423f14af7be4d8e7ffbd218fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.972119 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad21fd1f5d1f45fb99a0560e3405d69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.972890 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 776a235a8055419e975008f6c607ca57 (DEFAULT_TABLE table=default.Table_8da6aee6329246318cec0d90763e6dde [id=30533d5803cf420197caceaa22294ca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.973515 1475 tablet_service.cc:1505] Processing CreateTablet for tablet ad21fd1f5d1f45fb99a0560e3405d69f (DEFAULT_TABLE table=default.Table_01c58de0bf6845b08ff6fab88827cdec [id=4cdc1ba423f14af7be4d8e7ffbd218fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.973691 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 3c708051e9754fe8afc45154b0a4f30c (DEFAULT_TABLE table=default.Table_3e8db9d6df6542088fef2a3b687daddd [id=df27e3eb9ee648aa9825431915fcd8af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.974053 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 563897855a6e4c23803bf4ae34115850 (DEFAULT_TABLE table=default.Table_e2e3a2eac58e42fa8d80c405e3ec136a [id=db4af0d9193f45118561b3bd78834210]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.974135 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 563897855a6e4c23803bf4ae34115850. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.974392 1482 tablet_service.cc:1505] Processing CreateTablet for tablet f953a1b7d93743a5b1cdcbe9bc164923 (DEFAULT_TABLE table=default.Table_86bb3105657540d49403f9b2791e1134 [id=17daaf93b7ac4f8090683b6929098d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.974750 1477 tablet_service.cc:1505] Processing CreateTablet for tablet b8d30b2fdeec410fa582a05db0968ebb (DEFAULT_TABLE table=default.Table_9bc761287e4a44ac958794e2ba5b7b79 [id=b6210ccdd78245f09ae700c7907b427c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.975106 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 6e419003a8644b0dbd4ce325a0b49d1e (DEFAULT_TABLE table=default.Table_dbfe837064fb41238410f42ada6b39a7 [id=6244042d94f442bf953c891d5690ae78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.975499 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 3cdd85a3ca7545eea953e149a6c0d572 (DEFAULT_TABLE table=default.Table_0dd268e924c14ae1995b078b85afca08 [id=b1b6eae1d691498c8d5ca1f837ccc288]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.976106 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af918843c74940888c709d9a05aa6e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.977834 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c708051e9754fe8afc45154b0a4f30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.978493 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e419003a8644b0dbd4ce325a0b49d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.981019 1768 raft_consensus.cc:385] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:15.981305 1768 raft_consensus.cc:740] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:15.951186 1344 tablet_service.cc:1505] Processing CreateTablet for tablet af918843c74940888c709d9a05aa6e0f (DEFAULT_TABLE table=default.Table_ec77ed043045411b85ba06840af39907 [id=a6c04f5524384b75bfcedd008167716b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.952042 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 563897855a6e4c23803bf4ae34115850 (DEFAULT_TABLE table=default.Table_e2e3a2eac58e42fa8d80c405e3ec136a [id=db4af0d9193f45118561b3bd78834210]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.958637 1339 tablet_service.cc:1505] Processing CreateTablet for tablet f953a1b7d93743a5b1cdcbe9bc164923 (DEFAULT_TABLE table=default.Table_86bb3105657540d49403f9b2791e1134 [id=17daaf93b7ac4f8090683b6929098d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.958997 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 776a235a8055419e975008f6c607ca57 (DEFAULT_TABLE table=default.Table_8da6aee6329246318cec0d90763e6dde [id=30533d5803cf420197caceaa22294ca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.959358 1337 tablet_service.cc:1505] Processing CreateTablet for tablet b8d30b2fdeec410fa582a05db0968ebb (DEFAULT_TABLE table=default.Table_9bc761287e4a44ac958794e2ba5b7b79 [id=b6210ccdd78245f09ae700c7907b427c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.959697 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 6e419003a8644b0dbd4ce325a0b49d1e (DEFAULT_TABLE table=default.Table_dbfe837064fb41238410f42ada6b39a7 [id=6244042d94f442bf953c891d5690ae78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.960634 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 3cdd85a3ca7545eea953e149a6c0d572 (DEFAULT_TABLE table=default.Table_0dd268e924c14ae1995b078b85afca08 [id=b1b6eae1d691498c8d5ca1f837ccc288]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.960983 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 3c708051e9754fe8afc45154b0a4f30c (DEFAULT_TABLE table=default.Table_3e8db9d6df6542088fef2a3b687daddd [id=df27e3eb9ee648aa9825431915fcd8af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.971169 1343 tablet_service.cc:1505] Processing CreateTablet for tablet ad21fd1f5d1f45fb99a0560e3405d69f (DEFAULT_TABLE table=default.Table_01c58de0bf6845b08ff6fab88827cdec [id=4cdc1ba423f14af7be4d8e7ffbd218fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.976716 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 265040336fb347cca1d52dd331a1418d (DEFAULT_TABLE table=default.Table_b2be2ca906f34c9c855f39abd4548bf0 [id=e536d5403bb849af942f7c15e50989f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.982404 1617 tablet_service.cc:1505] Processing CreateTablet for tablet a4836025de6e4f51996034eb19002ba7 (DEFAULT_TABLE table=default.Table_d6276fea31c74f33b4f8c1258ec7b0a6 [id=82b538a5c4b147f3b93736528c234e4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.978686 1349 tablet_service.cc:1505] Processing CreateTablet for tablet a4836025de6e4f51996034eb19002ba7 (DEFAULT_TABLE table=default.Table_d6276fea31c74f33b4f8c1258ec7b0a6 [id=82b538a5c4b147f3b93736528c234e4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.982712 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4836025de6e4f51996034eb19002ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.983054 1333 tablet_service.cc:1505] Processing CreateTablet for tablet d123756f164f44168200b60bf7c615ca (DEFAULT_TABLE table=default.Table_f04fa9e7d054461894b1ded30ab195a1 [id=a8021b744dac410892c7ad1696598be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.977830 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 265040336fb347cca1d52dd331a1418d (DEFAULT_TABLE table=default.Table_b2be2ca906f34c9c855f39abd4548bf0 [id=e536d5403bb849af942f7c15e50989f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:15.986083 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 265040336fb347cca1d52dd331a1418d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.989948 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776a235a8055419e975008f6c607ca57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.990180 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 563897855a6e4c23803bf4ae34115850. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.990434 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3887fc21726846279fecb02672b9bc07" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:15.990617 1505 raft_consensus.cc:2468] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:15.991706 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e419003a8644b0dbd4ce325a0b49d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.992331 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8d30b2fdeec410fa582a05db0968ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.992573 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cdd85a3ca7545eea953e149a6c0d572. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.994923 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c708051e9754fe8afc45154b0a4f30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.000396 1767 tablet_bootstrap.cc:492] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:16.000605 1767 ts_tablet_manager.cc:1403] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.184s user 0.003s sys 0.002s
I20251024 19:48:16.001735 1767 raft_consensus.cc:359] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.001976 1767 raft_consensus.cc:385] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.002076 1767 raft_consensus.cc:740] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.002337 1767 consensus_queue.cc:260] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.002955 1767 ts_tablet_manager.cc:1434] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:16.003268 1767 tablet_bootstrap.cc:492] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:16.005522 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad21fd1f5d1f45fb99a0560e3405d69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.006064 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f953a1b7d93743a5b1cdcbe9bc164923. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.007608 1479 tablet_service.cc:1505] Processing CreateTablet for tablet a4836025de6e4f51996034eb19002ba7 (DEFAULT_TABLE table=default.Table_d6276fea31c74f33b4f8c1258ec7b0a6 [id=82b538a5c4b147f3b93736528c234e4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.008299 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 265040336fb347cca1d52dd331a1418d (DEFAULT_TABLE table=default.Table_b2be2ca906f34c9c855f39abd4548bf0 [id=e536d5403bb849af942f7c15e50989f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.008709 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4836025de6e4f51996034eb19002ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.010660 2561 consensus_queue.cc:1048] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:16.012621 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 265040336fb347cca1d52dd331a1418d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.017926 1489 tablet_service.cc:1505] Processing CreateTablet for tablet d123756f164f44168200b60bf7c615ca (DEFAULT_TABLE table=default.Table_f04fa9e7d054461894b1ded30ab195a1 [id=a8021b744dac410892c7ad1696598be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.021049 1766 tablet_bootstrap.cc:654] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.023013 1767 tablet_bootstrap.cc:654] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.023952 1557 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:16.024324 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d123756f164f44168200b60bf7c615ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:15.986090 1611 tablet_service.cc:1505] Processing CreateTablet for tablet d123756f164f44168200b60bf7c615ca (DEFAULT_TABLE table=default.Table_f04fa9e7d054461894b1ded30ab195a1 [id=a8021b744dac410892c7ad1696598be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.025072 1775 ts_tablet_manager.cc:1929] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.025183 1775 log.cc:1199] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/fdfa5eea3b674a3389a156d1c72204df
I20251024 19:48:16.025547 1775 ts_tablet_manager.cc:1950] T fdfa5eea3b674a3389a156d1c72204df P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.028302 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 694) took 4633 ms. Trace:
I20251024 19:48:16.029018 1586 leader_election.cc:304] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:16.028596 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d123756f164f44168200b60bf7c615ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.030767 2923 raft_consensus.cc:2804] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:16.030853 2923 raft_consensus.cc:493] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:16.030922 2923 raft_consensus.cc:3060] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.031818 1776 ts_tablet_manager.cc:1929] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.031917 1776 log.cc:1199] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/cef80e4d14bb4e719bceebf43041d079
I20251024 19:48:16.032258 1776 ts_tablet_manager.cc:1950] T cef80e4d14bb4e719bceebf43041d079 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:16.028401 1775 rpcz_store.cc:276] 1024 19:48:11.394790 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.394843 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:15.892177 (+4497334us) ts_tablet_manager.cc:1151] Deleting tablet fdfa5eea3b674a3389a156d1c72204df
1024 19:48:15.892187 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.922925 (+ 30738us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.024963 (+102038us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.028281 (+ 3318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4497196}
I20251024 19:48:16.038347 1775 tablet_replica.cc:333] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:16.038758 1775 raft_consensus.cc:2243] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:16.039135 1775 raft_consensus.cc:2272] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.047392 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet fdfa5eea3b674a3389a156d1c72204df (table default.Table_73923b0648674b2891cbbf4b90d3f106 [id=96ea5a86f92b4d18b0d74f1092bd17b3]) successfully deleted
I20251024 19:48:16.046829 1767 tablet_bootstrap.cc:492] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:16.047714 1767 ts_tablet_manager.cc:1403] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.001s
I20251024 19:48:16.049512 1767 raft_consensus.cc:359] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.049892 1767 raft_consensus.cc:385] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.050019 1767 raft_consensus.cc:740] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.050518 1767 consensus_queue.cc:260] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.050177 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 747) took 4068 ms. Trace:
I20251024 19:48:16.051214 1776 rpcz_store.cc:276] 1024 19:48:11.980923 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.981109 (+ 186us) service_pool.cc:225] Handling call
1024 19:48:15.899669 (+3918560us) ts_tablet_manager.cc:1151] Deleting tablet cef80e4d14bb4e719bceebf43041d079
1024 19:48:15.899682 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:15.988280 (+ 88598us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.031724 (+ 43444us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.049838 (+ 18114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3918122}
I20251024 19:48:16.051244 1767 ts_tablet_manager.cc:1434] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:16.053977 1767 tablet_bootstrap.cc:492] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:16.054519 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet cef80e4d14bb4e719bceebf43041d079 (table default.Table_ade7993306ee438baebbf639e2109768 [id=6318d0634b77498a8c1ff8618b8ccf4b]) successfully deleted
I20251024 19:48:16.055771 2923 raft_consensus.cc:515] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.039695 1775 ts_tablet_manager.cc:1916] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.056258 1776 tablet_replica.cc:333] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:16.060016 1767 tablet_bootstrap.cc:654] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.094848 1776 raft_consensus.cc:2243] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.095052 1776 raft_consensus.cc:2272] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.095626 1776 ts_tablet_manager.cc:1916] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.103961 1776 ts_tablet_manager.cc:1929] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251024 19:48:16.106367 1447 consensus_peers.cc:597] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:16.148340 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3887fc21726846279fecb02672b9bc07" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:16.149369 1508 raft_consensus.cc:3060] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.153054 1508 raft_consensus.cc:2468] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
W20251024 19:48:16.165324 1447 consensus_peers.cc:597] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a7e76a3cac443ec8d2ab466075aae6b. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:16.171990 1766 tablet_bootstrap.cc:492] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:16.172135 1766 ts_tablet_manager.cc:1403] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.230s user 0.007s sys 0.000s
I20251024 19:48:16.172887 1766 raft_consensus.cc:359] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.173054 1766 raft_consensus.cc:385] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.173105 1766 raft_consensus.cc:740] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.173290 1766 consensus_queue.cc:260] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.175211 1766 ts_tablet_manager.cc:1434] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251024 19:48:16.175781 1766 tablet_bootstrap.cc:492] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:16.181614 2561 raft_consensus.cc:493] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.190721 2561 raft_consensus.cc:515] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.105445 1767 tablet_bootstrap.cc:492] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:16.194151 1776 log.cc:1199] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/701c93a600964c20a275f5942a2ff863
I20251024 19:48:16.193529 1767 ts_tablet_manager.cc:1403] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.140s user 0.004s sys 0.000s
I20251024 19:48:16.196173 1767 raft_consensus.cc:359] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.196357 1767 raft_consensus.cc:385] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.196412 1767 raft_consensus.cc:740] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.196740 1767 consensus_queue.cc:260] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.197325 1767 ts_tablet_manager.cc:1434] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:16.198477 1767 tablet_bootstrap.cc:492] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:16.200502 1767 tablet_bootstrap.cc:654] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.107226 2923 leader_election.cc:290] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:16.141511 2569 raft_consensus.cc:493] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.201819 2569 raft_consensus.cc:515] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.202500 2569 leader_election.cc:290] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:16.153640 1586 leader_election.cc:304] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:16.202728 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 49158us
I20251024 19:48:16.203189 2920 raft_consensus.cc:2804] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:16.203301 2920 raft_consensus.cc:697] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:16.203565 2920 consensus_queue.cc:237] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.214546 2561 leader_election.cc:290] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:16.216451 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c82f60819e7a442bbc0cc04659e14e87" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:16.217648 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c82f60819e7a442bbc0cc04659e14e87" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:16.217941 1644 raft_consensus.cc:2468] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:16.218410 1444 leader_election.cc:304] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:16.218724 2885 raft_consensus.cc:2804] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:16.218803 2885 raft_consensus.cc:493] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:16.218870 2885 raft_consensus.cc:3060] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.221253 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c82f60819e7a442bbc0cc04659e14e87" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:16.229077 2885 raft_consensus.cc:515] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.229887 1508 raft_consensus.cc:2393] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
I20251024 19:48:16.230861 2885 leader_election.cc:290] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:16.231549 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c82f60819e7a442bbc0cc04659e14e87" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:16.231729 1643 raft_consensus.cc:3060] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.234769 1643 raft_consensus.cc:2468] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:16.235318 1444 leader_election.cc:304] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:16.236467 2936 raft_consensus.cc:493] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.236615 2936 raft_consensus.cc:515] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.237290 2936 leader_election.cc:290] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:16.237838 2808 raft_consensus.cc:2804] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:16.237960 2808 raft_consensus.cc:697] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:16.238202 2808 consensus_queue.cc:237] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.242270 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce8c179e98944df0b510a0e8daf3920f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:16.242486 1643 raft_consensus.cc:2468] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:16.242964 1305 leader_election.cc:304] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:16.243268 2936 raft_consensus.cc:2804] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:16.243350 2936 raft_consensus.cc:493] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:16.243422 2936 raft_consensus.cc:3060] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.243638 1768 consensus_queue.cc:260] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.244822 1768 ts_tablet_manager.cc:1434] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.307s user 0.003s sys 0.000s
I20251024 19:48:16.246502 2936 raft_consensus.cc:515] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.247692 2936 leader_election.cc:290] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:16.250859 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce8c179e98944df0b510a0e8daf3920f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:16.251479 1643 raft_consensus.cc:3060] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.254009 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af918843c74940888c709d9a05aa6e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.255729 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f953a1b7d93743a5b1cdcbe9bc164923. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.258641 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8d30b2fdeec410fa582a05db0968ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.259481 1774 ts_tablet_manager.cc:1929] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.4
I20251024 19:48:16.259584 1774 log.cc:1199] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/452f77091adc4cf09de2231f2f6d845b
I20251024 19:48:16.259933 1774 ts_tablet_manager.cc:1950] T 452f77091adc4cf09de2231f2f6d845b P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:16.261138 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d123756f164f44168200b60bf7c615ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.262529 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cdd85a3ca7545eea953e149a6c0d572. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.264915 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e419003a8644b0dbd4ce325a0b49d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.266162 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 664) took 4454 ms. Trace:
I20251024 19:48:16.266412 1774 rpcz_store.cc:276] 1024 19:48:11.811765 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.812253 (+ 488us) service_pool.cc:225] Handling call
1024 19:48:15.882467 (+4070214us) ts_tablet_manager.cc:1151] Deleting tablet 452f77091adc4cf09de2231f2f6d845b
1024 19:48:15.882477 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.067898 (+185421us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.259389 (+191491us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.266139 (+ 6750us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4070053}
I20251024 19:48:16.267266 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 682) took 4221 ms. Trace:
I20251024 19:48:16.267277 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 452f77091adc4cf09de2231f2f6d845b (table default.Table_5bfbf5d7f5764b96a95bf3ac0e0aba69 [id=9a49120c5b654676a63cd3966eaf31a9]) successfully deleted
I20251024 19:48:16.267444 1774 rpcz_store.cc:276] 1024 19:48:12.045557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.045928 (+ 371us) service_pool.cc:225] Handling call
1024 19:48:16.267079 (+4221151us) ts_tablet_manager.cc:1151] Deleting tablet 64d8da45c6364f1e9ddc79307904587c
1024 19:48:16.267090 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.267230 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4220878}
W20251024 19:48:16.268142 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 64d8da45c6364f1e9ddc79307904587c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d8da45c6364f1e9ddc79307904587c already in progress: creating tablet
I20251024 19:48:16.269088 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 683) took 4223 ms. Trace:
I20251024 19:48:16.266728 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 265040336fb347cca1d52dd331a1418d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.269196 1774 rpcz_store.cc:276] 1024 19:48:12.045757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.046213 (+ 456us) service_pool.cc:225] Handling call
1024 19:48:16.268505 (+4222292us) ts_tablet_manager.cc:1151] Deleting tablet 844638a6b6944d0394df3e78a8ea51d6
1024 19:48:16.268979 (+ 474us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.269070 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4222154}
I20251024 19:48:16.269515 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 681) took 4224 ms. Trace:
W20251024 19:48:16.269667 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 844638a6b6944d0394df3e78a8ea51d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 844638a6b6944d0394df3e78a8ea51d6 already in progress: creating tablet
I20251024 19:48:16.269621 1774 rpcz_store.cc:276] 1024 19:48:12.045355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.046366 (+ 1011us) service_pool.cc:225] Handling call
1024 19:48:16.269424 (+4223058us) ts_tablet_manager.cc:1151] Deleting tablet eee675874516489f9e8dac41f70ac4e4
1024 19:48:16.269432 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.269500 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4222842}
I20251024 19:48:16.269816 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776a235a8055419e975008f6c607ca57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.269912 1774 tablet_replica.cc:333] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:16.270079 1774 raft_consensus.cc:2243] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.270255 1774 raft_consensus.cc:2272] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:16.270398 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet eee675874516489f9e8dac41f70ac4e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eee675874516489f9e8dac41f70ac4e4 already in progress: creating tablet
I20251024 19:48:16.270658 2561 raft_consensus.cc:493] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.270819 2561 raft_consensus.cc:515] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.271514 1774 ts_tablet_manager.cc:1916] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.271873 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4836025de6e4f51996034eb19002ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.272287 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 563897855a6e4c23803bf4ae34115850. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.272717 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3887fc21726846279fecb02672b9bc07" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:16.273190 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3887fc21726846279fecb02672b9bc07" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:16.275195 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad21fd1f5d1f45fb99a0560e3405d69f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.283306 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c708051e9754fe8afc45154b0a4f30c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:16.293977 1447 leader_election.cc:343] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.295831 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c82f60819e7a442bbc0cc04659e14e87" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:16.296106 2561 leader_election.cc:290] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:16.296618 1447 leader_election.cc:343] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.296795 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9489a01434a24d9ca94791252950725c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:16.296844 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9489a01434a24d9ca94791252950725c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:16.297303 1444 leader_election.cc:343] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:16.297469 1447 leader_election.cc:343] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.297729 1447 leader_election.cc:304] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:16.298004 1766 tablet_bootstrap.cc:654] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.298254 2561 raft_consensus.cc:2749] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:16.311286 2923 raft_consensus.cc:493] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.311920 2923 raft_consensus.cc:515] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:16.312567 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4421b15a60d4256aeaa894bb1e4f99d consensus state has no leader
I20251024 19:48:16.316514 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639836c656f44a819036f4d00e055bd9" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:16.316721 1508 raft_consensus.cc:2468] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:16.320042 1586 leader_election.cc:304] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:16.320360 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c82f60819e7a442bbc0cc04659e14e87" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:16.320771 2920 raft_consensus.cc:2804] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:16.320854 2920 raft_consensus.cc:493] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:16.321018 2920 raft_consensus.cc:3060] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.321725 2923 leader_election.cc:290] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:16.323094 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e83527f7ff5c4e248f344a84515a5d6a consensus state has no leader
I20251024 19:48:16.338040 2937 tablet.cc:1722] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:16.340853 2920 raft_consensus.cc:515] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.342252 1766 tablet_bootstrap.cc:492] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:16.342424 1766 ts_tablet_manager.cc:1403] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.167s user 0.003s sys 0.000s
I20251024 19:48:16.342901 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639836c656f44a819036f4d00e055bd9" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:16.342999 2920 leader_election.cc:290] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:16.343680 1766 raft_consensus.cc:359] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.343895 1766 raft_consensus.cc:385] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.343958 1766 raft_consensus.cc:740] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.343701 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639836c656f44a819036f4d00e055bd9" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:16.344148 1766 consensus_queue.cc:260] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.344275 1508 raft_consensus.cc:3060] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.344693 1766 ts_tablet_manager.cc:1434] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:16.345010 1766 tablet_bootstrap.cc:492] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:16.345072 1643 raft_consensus.cc:2468] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:16.345523 1305 leader_election.cc:304] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:16.346091 2936 raft_consensus.cc:2804] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:16.346729 1766 tablet_bootstrap.cc:654] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.347656 2936 raft_consensus.cc:697] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:16.347771 1508 raft_consensus.cc:2468] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:16.348284 1586 leader_election.cc:304] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:16.348114 2936 consensus_queue.cc:237] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.349792 2923 consensus_queue.cc:1048] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:16.350458 2923 raft_consensus.cc:2804] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:16.351020 2923 raft_consensus.cc:697] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:16.351325 2923 consensus_queue.cc:237] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.352094 1696 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:16.359246 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639836c656f44a819036f4d00e055bd9" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:16.360025 1586 consensus_peers.cc:597] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:16.359589 2976 tablet.cc:1722] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251024 19:48:16.360620 1586 leader_election.cc:343] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.363528 1776 ts_tablet_manager.cc:1950] T 701c93a600964c20a275f5942a2ff863 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:16.363906 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 746) took 4383 ms. Trace:
I20251024 19:48:16.364042 1776 rpcz_store.cc:276] 1024 19:48:11.980822 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.982082 (+ 1260us) service_pool.cc:225] Handling call
1024 19:48:16.055510 (+4073428us) ts_tablet_manager.cc:1151] Deleting tablet 701c93a600964c20a275f5942a2ff863
1024 19:48:16.055526 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.100369 (+ 44843us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.103705 (+ 3336us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.363887 (+260182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1332,"tablet-delete.queue_time_us":4072847}
I20251024 19:48:16.364673 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 701c93a600964c20a275f5942a2ff863 (table default.Table_7dc53b918ca7443b807133df217c0f75 [id=9ce6ac48924e45e3a482f71740edb4c0]) successfully deleted
I20251024 19:48:16.365101 1776 tablet_replica.cc:333] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:48:16.365938 1586 leader_election.cc:343] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:16.366392 1447 consensus_peers.cc:597] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 452f77091adc4cf09de2231f2f6d845b. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:16.367782 1586 leader_election.cc:343] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.367946 1586 leader_election.cc:304] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:16.368400 2569 raft_consensus.cc:2749] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:16.370699 1767 tablet_bootstrap.cc:492] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:16.370898 1767 ts_tablet_manager.cc:1403] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.173s user 0.007s sys 0.003s
W20251024 19:48:16.371510 1586 consensus_peers.cc:597] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:16.372221 1586 leader_election.cc:343] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.372308 1767 raft_consensus.cc:359] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.372519 1767 raft_consensus.cc:385] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.372625 1767 raft_consensus.cc:740] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.372931 1767 consensus_queue.cc:260] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.373634 1767 ts_tablet_manager.cc:1434] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251024 19:48:16.375766 1370 raft_consensus.cc:3060] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:16.389451 1308 leader_election.cc:343] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: ce8c179e98944df0b510a0e8daf3920f
I20251024 19:48:16.389912 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f76a1631002d4ee0a3323bcbd2e339d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:16.390581 2776 log.cc:927] Time spent T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Append to log took a long time: real 0.098s user 0.003s sys 0.000s
W20251024 19:48:16.390617 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f76a1631002d4ee0a3323bcbd2e339d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:16.393766 1308 leader_election.cc:343] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: ce8c179e98944df0b510a0e8daf3920f
I20251024 19:48:16.397240 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_420ebc050cf440e9a59a18e26f7b83af" } modify_external_catalogs: true
I20251024 19:48:16.397351 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fa23a3c70c3247f9ac84e69f79a41640" } new_table_name: "default.Table_ea78d03887214fd59d5ce1c5e137a7e0" modify_external_catalogs: true
I20251024 19:48:16.397475 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_420ebc050cf440e9a59a18e26f7b83af" } modify_external_catalogs: true
I20251024 19:48:16.400003 1775 ts_tablet_manager.cc:1929] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.400149 1775 log.cc:1199] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/701c93a600964c20a275f5942a2ff863
I20251024 19:48:16.400540 1775 ts_tablet_manager.cc:1950] T 701c93a600964c20a275f5942a2ff863 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.400869 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 695) took 4888 ms. Trace:
I20251024 19:48:16.400957 1775 rpcz_store.cc:276] 1024 19:48:11.512196 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.512286 (+ 90us) service_pool.cc:225] Handling call
1024 19:48:16.038298 (+4526012us) ts_tablet_manager.cc:1151] Deleting tablet 701c93a600964c20a275f5942a2ff863
1024 19:48:16.038308 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.169702 (+131394us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.399882 (+230180us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.400852 (+ 970us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4525720}
I20251024 19:48:16.401718 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 701c93a600964c20a275f5942a2ff863 (table default.Table_7dc53b918ca7443b807133df217c0f75 [id=9ce6ac48924e45e3a482f71740edb4c0]) successfully deleted
I20251024 19:48:16.402377 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 704) took 4847 ms. Trace:
I20251024 19:48:16.400545 1774 ts_tablet_manager.cc:1929] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.403086 1774 log.cc:1199] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/dc14b0cdb7f342ac956a7857ab3f35d7
I20251024 19:48:16.404412 1774 ts_tablet_manager.cc:1950] T dc14b0cdb7f342ac956a7857ab3f35d7 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:16.406081 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 676) took 4466 ms. Trace:
I20251024 19:48:16.408969 1774 rpcz_store.cc:276] 1024 19:48:11.939265 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.075696 (+136431us) service_pool.cc:225] Handling call
1024 19:48:16.269871 (+4194175us) ts_tablet_manager.cc:1151] Deleting tablet dc14b0cdb7f342ac956a7857ab3f35d7
1024 19:48:16.269880 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.278063 (+ 8183us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.320051 (+ 41988us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.406064 (+ 86013us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4193909}
I20251024 19:48:16.414350 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet dc14b0cdb7f342ac956a7857ab3f35d7 (table default.Table_be61dd09c4c94f568d723c4bace3a038 [id=66be3fc672ea4a9e9fb5dde3ae5e5b62]) successfully deleted
I20251024 19:48:16.415024 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 686) took 4204 ms. Trace:
W20251024 19:48:16.388777 1586 leader_election.cc:343] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:16.448385 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 59620.2us
I20251024 19:48:16.449863 1767 tablet_bootstrap.cc:492] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:16.475611 1774 rpcz_store.cc:276] 1024 19:48:12.210865 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.210984 (+ 119us) service_pool.cc:225] Handling call
1024 19:48:16.414881 (+4203897us) ts_tablet_manager.cc:1151] Deleting tablet 373aa543e1284fbc945ca802d9eddf59
1024 19:48:16.414892 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.415003 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4136042}
I20251024 19:48:16.392975 1776 raft_consensus.cc:2243] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:16.476361 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 373aa543e1284fbc945ca802d9eddf59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 373aa543e1284fbc945ca802d9eddf59 already in progress: creating tablet
I20251024 19:48:16.477044 1774 tablet_replica.cc:333] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:16.477334 1774 raft_consensus.cc:2243] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.420408 1768 tablet_bootstrap.cc:492] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:16.480830 1768 tablet_bootstrap.cc:654] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.488914 1776 raft_consensus.cc:2272] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.489522 1776 ts_tablet_manager.cc:1916] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:16.453907 1444 consensus_peers.cc:597] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cef80e4d14bb4e719bceebf43041d079. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:16.493836 1775 rpcz_store.cc:276] 1024 19:48:11.555167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.555219 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:16.402246 (+4847027us) ts_tablet_manager.cc:1151] Deleting tablet 82cf75ee148e45b29da3343b3b068781
1024 19:48:16.402258 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.402359 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4846867}
W20251024 19:48:16.494359 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 82cf75ee148e45b29da3343b3b068781 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82cf75ee148e45b29da3343b3b068781 already in progress: creating tablet
I20251024 19:48:16.494446 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 707) took 4928 ms. Trace:
I20251024 19:48:16.494514 1775 rpcz_store.cc:276] 1024 19:48:11.566093 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.571845 (+ 5752us) service_pool.cc:225] Handling call
1024 19:48:16.494302 (+4922457us) ts_tablet_manager.cc:1151] Deleting tablet 8876b7b38e744212a3722dbe5bcc5596
1024 19:48:16.494316 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.494425 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4922290}
I20251024 19:48:16.494947 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 709) took 4918 ms. Trace:
W20251024 19:48:16.495010 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 8876b7b38e744212a3722dbe5bcc5596 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8876b7b38e744212a3722dbe5bcc5596 already in progress: creating tablet
I20251024 19:48:16.495028 1775 rpcz_store.cc:276] 1024 19:48:11.576158 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.576215 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:16.494844 (+4918629us) ts_tablet_manager.cc:1151] Deleting tablet cb0405765c6146cbbb187af96a60be2e
1024 19:48:16.494852 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.494932 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4918473}
W20251024 19:48:16.495420 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet cb0405765c6146cbbb187af96a60be2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb0405765c6146cbbb187af96a60be2e already in progress: creating tablet
I20251024 19:48:16.495424 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 711) took 4826 ms. Trace:
I20251024 19:48:16.495520 1775 rpcz_store.cc:276] 1024 19:48:11.669126 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.669188 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:16.495326 (+4826138us) ts_tablet_manager.cc:1151] Deleting tablet cbd321c059e34834a6257fab5e6d14bb
1024 19:48:16.495335 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.495408 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4825869}
W20251024 19:48:16.495857 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet cbd321c059e34834a6257fab5e6d14bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbd321c059e34834a6257fab5e6d14bb already in progress: creating tablet
I20251024 19:48:16.497746 1774 raft_consensus.cc:2272] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.498379 1774 ts_tablet_manager.cc:1916] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.499738 1767 tablet_bootstrap.cc:654] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.500447 2985 tablet.cc:1722] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:16.501693 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 713) took 4824 ms. Trace:
I20251024 19:48:16.502856 1775 rpcz_store.cc:276] 1024 19:48:11.676894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.676965 (+ 71us) service_pool.cc:225] Handling call
1024 19:48:16.501564 (+4824599us) ts_tablet_manager.cc:1151] Deleting tablet 50cea450eda9485daba40bf875a1b90c
1024 19:48:16.501592 (+ 28us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.501676 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4824390}
W20251024 19:48:16.505957 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 50cea450eda9485daba40bf875a1b90c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50cea450eda9485daba40bf875a1b90c already in progress: creating tablet
I20251024 19:48:16.506381 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 712) took 4835 ms. Trace:
I20251024 19:48:16.508872 1775 rpcz_store.cc:276] 1024 19:48:11.676739 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.677221 (+ 482us) service_pool.cc:225] Handling call
1024 19:48:16.506282 (+4829061us) ts_tablet_manager.cc:1151] Deleting tablet 6f7b6d15f25d4c2a905da6597ba8d98a
1024 19:48:16.506291 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.506366 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4828768}
I20251024 19:48:16.509678 1775 tablet_replica.cc:333] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:16.509964 1775 raft_consensus.cc:2243] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 3 LEADER]: Raft consensus shutting down.
I20251024 19:48:16.510383 1775 raft_consensus.cc:2272] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e [term 3 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.510998 1775 ts_tablet_manager.cc:1916] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.514516 1766 tablet_bootstrap.cc:492] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:16.514670 1766 ts_tablet_manager.cc:1403] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.170s user 0.030s sys 0.037s
I20251024 19:48:16.515612 1766 raft_consensus.cc:359] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.515821 1766 raft_consensus.cc:385] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.515892 1766 raft_consensus.cc:740] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.516139 1766 consensus_queue.cc:260] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.516739 1766 ts_tablet_manager.cc:1434] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:16.517107 1766 tablet_bootstrap.cc:492] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:16.518904 1766 tablet_bootstrap.cc:654] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.530368 1776 ts_tablet_manager.cc:1929] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.530652 1768 tablet_bootstrap.cc:492] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:16.530912 1768 ts_tablet_manager.cc:1403] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.111s user 0.003s sys 0.000s
I20251024 19:48:16.531318 1767 tablet_bootstrap.cc:492] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:16.531515 1767 ts_tablet_manager.cc:1403] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.000s
I20251024 19:48:16.531725 1768 raft_consensus.cc:359] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.532248 1768 raft_consensus.cc:385] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.532336 1767 raft_consensus.cc:359] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.532444 1768 raft_consensus.cc:740] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.532589 1767 raft_consensus.cc:385] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.532718 1767 raft_consensus.cc:740] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.532891 1768 consensus_queue.cc:260] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.533004 1767 consensus_queue.cc:260] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.533592 1767 ts_tablet_manager.cc:1434] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:16.537170 1766 tablet_bootstrap.cc:492] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:16.537330 1766 ts_tablet_manager.cc:1403] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.007s
I20251024 19:48:16.538369 1766 raft_consensus.cc:359] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.538574 1766 raft_consensus.cc:385] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.538632 1766 raft_consensus.cc:740] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.538837 1766 consensus_queue.cc:260] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.539582 1766 ts_tablet_manager.cc:1434] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251024 19:48:16.539955 1766 tablet_bootstrap.cc:492] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:16.541666 1767 tablet_bootstrap.cc:492] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:16.543510 1767 tablet_bootstrap.cc:654] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.543686 1776 log.cc:1199] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/b88a6a5237d146f5b6f082ca3af25b2b
I20251024 19:48:16.544893 1776 ts_tablet_manager.cc:1950] T b88a6a5237d146f5b6f082ca3af25b2b P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:16.546737 1766 tablet_bootstrap.cc:654] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.554824 2920 consensus_queue.cc:1048] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:16.562121 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 745) took 4581 ms. Trace:
I20251024 19:48:16.562203 1766 tablet_bootstrap.cc:492] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:16.562327 1766 ts_tablet_manager.cc:1403] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.001s
I20251024 19:48:16.562230 1776 rpcz_store.cc:276] 1024 19:48:11.980720 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.982619 (+ 1899us) service_pool.cc:225] Handling call
1024 19:48:16.365037 (+4382418us) ts_tablet_manager.cc:1151] Deleting tablet b88a6a5237d146f5b6f082ca3af25b2b
1024 19:48:16.365052 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.492746 (+127694us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.530249 (+ 37503us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.562100 (+ 31851us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32128,"tablet-delete.queue_time_us":4382181}
I20251024 19:48:16.562680 1776 tablet_replica.cc:333] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:16.562716 1768 ts_tablet_manager.cc:1434] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20251024 19:48:16.562875 1776 raft_consensus.cc:2243] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.562887 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet b88a6a5237d146f5b6f082ca3af25b2b (table default.Table_6eb10ae51f3a43c2ab15f362708fba94 [id=f3080df28ee94876bb6b790cea45a4e8]) successfully deleted
I20251024 19:48:16.563203 1768 tablet_bootstrap.cc:492] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:16.564590 1776 raft_consensus.cc:2272] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.565135 1776 ts_tablet_manager.cc:1916] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.565702 1766 raft_consensus.cc:359] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.565855 1766 raft_consensus.cc:385] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.565898 1766 raft_consensus.cc:740] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.566135 1766 consensus_queue.cc:260] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.566599 1775 ts_tablet_manager.cc:1929] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.4
I20251024 19:48:16.566708 1766 ts_tablet_manager.cc:1434] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251024 19:48:16.567035 1766 tablet_bootstrap.cc:492] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:16.567953 1418 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:16.569253 1766 tablet_bootstrap.cc:654] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.566717 1775 log.cc:1199] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/452f77091adc4cf09de2231f2f6d845b
I20251024 19:48:16.570636 1775 ts_tablet_manager.cc:1950] T 452f77091adc4cf09de2231f2f6d845b P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.570991 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 715) took 4819 ms. Trace:
I20251024 19:48:16.571080 1775 rpcz_store.cc:276] 1024 19:48:11.751541 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.751619 (+ 78us) service_pool.cc:225] Handling call
1024 19:48:16.509587 (+4757968us) ts_tablet_manager.cc:1151] Deleting tablet 452f77091adc4cf09de2231f2f6d845b
1024 19:48:16.509596 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.521698 (+ 12102us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.566495 (+ 44797us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.570970 (+ 4475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4757734}
I20251024 19:48:16.571437 1775 tablet_replica.cc:333] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:16.571630 1775 raft_consensus.cc:2243] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.571602 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 452f77091adc4cf09de2231f2f6d845b (table default.Table_5bfbf5d7f5764b96a95bf3ac0e0aba69 [id=9a49120c5b654676a63cd3966eaf31a9]) successfully deleted
I20251024 19:48:16.571780 1775 raft_consensus.cc:2272] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.572347 1775 ts_tablet_manager.cc:1916] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.572367 1767 tablet_bootstrap.cc:492] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:16.572510 1767 ts_tablet_manager.cc:1403] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20251024 19:48:16.573223 1767 raft_consensus.cc:359] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.573529 1767 raft_consensus.cc:385] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.573665 1767 raft_consensus.cc:740] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.573923 1767 consensus_queue.cc:260] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.574419 1767 ts_tablet_manager.cc:1434] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:16.576071 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 265040336fb347cca1d52dd331a1418d
I20251024 19:48:16.576813 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 265040336fb347cca1d52dd331a1418d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:16.577387 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2e6ab4e8ffb84bf0961af481f0d4b180
I20251024 19:48:16.578374 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 265040336fb347cca1d52dd331a1418d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:16.578495 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e6ab4e8ffb84bf0961af481f0d4b180 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:16.578871 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e6ab4e8ffb84bf0961af481f0d4b180 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:16.588652 1775 ts_tablet_manager.cc:1929] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.588766 1775 log.cc:1199] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/27b32126d04e480f8db7b2b20aba5873
I20251024 19:48:16.589147 1766 tablet_bootstrap.cc:492] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:16.589290 1766 ts_tablet_manager.cc:1403] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.002s
I20251024 19:48:16.590205 1766 raft_consensus.cc:359] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.590394 1766 raft_consensus.cc:385] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.590456 1766 raft_consensus.cc:740] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.590695 1766 consensus_queue.cc:260] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.594197 1768 tablet_bootstrap.cc:654] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.596391 1766 ts_tablet_manager.cc:1434] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251024 19:48:16.596958 1766 tablet_bootstrap.cc:492] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:16.598484 1768 tablet_bootstrap.cc:492] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:16.598548 1766 tablet_bootstrap.cc:654] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.599354 1768 ts_tablet_manager.cc:1403] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.001s
I20251024 19:48:16.600219 1768 raft_consensus.cc:359] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.589138 1775 ts_tablet_manager.cc:1950] T 27b32126d04e480f8db7b2b20aba5873 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.577356 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet 265040336fb347cca1d52dd331a1418d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:16.579216 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e6ab4e8ffb84bf0961af481f0d4b180 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:16.602377 2569 raft_consensus.cc:493] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.604653 2569 raft_consensus.cc:515] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.605340 2569 leader_election.cc:290] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:16.606318 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a4cb86bc5e745d5b4ee932f0a38b3c7" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:16.606504 1505 raft_consensus.cc:2468] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:16.606609 1767 tablet_bootstrap.cc:492] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:16.607040 1586 leader_election.cc:304] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:16.607118 1768 raft_consensus.cc:385] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.607523 2569 raft_consensus.cc:2804] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:16.607594 2569 raft_consensus.cc:493] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:16.607647 2569 raft_consensus.cc:3060] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.606863 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a4cb86bc5e745d5b4ee932f0a38b3c7" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:16.604300 1774 ts_tablet_manager.cc:1929] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.608062 1774 log.cc:1199] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/59c7dbe9ff2c4869994c4f43d5676c30
I20251024 19:48:16.608389 1774 ts_tablet_manager.cc:1950] T 59c7dbe9ff2c4869994c4f43d5676c30 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:16.609061 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 687) took 4352 ms. Trace:
I20251024 19:48:16.609133 1774 rpcz_store.cc:276] 1024 19:48:12.256947 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.260434 (+ 3487us) service_pool.cc:225] Handling call
1024 19:48:16.476904 (+4216470us) ts_tablet_manager.cc:1151] Deleting tablet 59c7dbe9ff2c4869994c4f43d5676c30
1024 19:48:16.476916 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.501539 (+ 24623us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.604190 (+102651us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.609043 (+ 4853us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4193983}
I20251024 19:48:16.609678 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 59c7dbe9ff2c4869994c4f43d5676c30 (table default.Table_eb9c8019451d46fba31a87a0b6ac6ad8 [id=e5b1dd31c2114e8a95ba33981029255c]) successfully deleted
I20251024 19:48:16.609890 1768 raft_consensus.cc:740] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.610375 2569 raft_consensus.cc:515] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.610953 2569 leader_election.cc:290] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:16.611086 2569 consensus_queue.cc:1048] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:16.612123 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a4cb86bc5e745d5b4ee932f0a38b3c7" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:16.612285 1508 raft_consensus.cc:3060] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.613806 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 714) took 4887 ms. Trace:
I20251024 19:48:16.613889 1775 rpcz_store.cc:276] 1024 19:48:11.726775 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.726924 (+ 149us) service_pool.cc:225] Handling call
1024 19:48:16.571385 (+4844461us) ts_tablet_manager.cc:1151] Deleting tablet 27b32126d04e480f8db7b2b20aba5873
1024 19:48:16.571395 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.576122 (+ 4727us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.588558 (+ 12436us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.613783 (+ 25225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4817176}
I20251024 19:48:16.614666 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a4cb86bc5e745d5b4ee932f0a38b3c7" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:16.614776 1586 leader_election.cc:343] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.615113 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 27b32126d04e480f8db7b2b20aba5873 (table default.Table_a372d22e91854206b3523ecd087303ca [id=dfffb9501a0549198d17b197ef05871e]) successfully deleted
W20251024 19:48:16.615355 1586 leader_election.cc:343] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.616866 1508 raft_consensus.cc:2468] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:16.619349 1775 tablet_replica.cc:333] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:16.619599 1775 raft_consensus.cc:2243] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:16.620011 1775 raft_consensus.cc:2272] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.620615 1775 ts_tablet_manager.cc:1916] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.621786 1586 leader_election.cc:304] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:16.622342 2920 raft_consensus.cc:2804] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:16.622545 2920 raft_consensus.cc:697] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:16.622915 2920 consensus_queue.cc:237] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.625252 1767 tablet_bootstrap.cc:654] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.627228 1766 tablet_bootstrap.cc:492] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:16.627375 1766 ts_tablet_manager.cc:1403] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.004s
W20251024 19:48:16.627498 1444 consensus_peers.cc:597] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b88a6a5237d146f5b6f082ca3af25b2b. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:16.628451 1766 raft_consensus.cc:359] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.628850 1766 raft_consensus.cc:385] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.628921 1766 raft_consensus.cc:740] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.629169 1766 consensus_queue.cc:260] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.629767 1766 ts_tablet_manager.cc:1434] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:16.630234 1766 tablet_bootstrap.cc:492] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:16.633652 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9026d9e801294dd2bfdb9c90e45a0990
I20251024 19:48:16.634325 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 9026d9e801294dd2bfdb9c90e45a0990 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:16.635303 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 9026d9e801294dd2bfdb9c90e45a0990 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:16.636003 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 9026d9e801294dd2bfdb9c90e45a0990 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:16.637297 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a712f52cd114e9bb07c2717e5d728fd consensus state has no leader
I20251024 19:48:16.638578 1183 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 4ffd446c269448d38cc8d987a53aecf9 (table default.Table_bedd8edc24014953ac8c7dcecf18f128 [id=2ae1c808837e48df97c4efb01a684da9]). Expected version 1 got 0
I20251024 19:48:16.639753 1187 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_e5c532eb7fc241039c7e06e4757ea011 [id=9065c680f4da42b5a6e5ca335c8e2f3b] alter complete (version 3)
I20251024 19:48:16.641227 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_727cead5e96a4e04af23383cb5503560" } modify_external_catalogs: true
I20251024 19:48:16.641676 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 41176885447e42409ee4363bb4d903f2
I20251024 19:48:16.642490 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet 41176885447e42409ee4363bb4d903f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:16 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:16.642522 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3864c444fa71452eb70094031c884901"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:16.642930 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 41176885447e42409ee4363bb4d903f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:16 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:16.643255 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3864c444fa71452eb70094031c884901 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:16.645150 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 41176885447e42409ee4363bb4d903f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:16 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:16.645632 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b841d8eb9cca40dcb3e45759fbf5acce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:16.646224 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b841d8eb9cca40dcb3e45759fbf5acce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:16.647814 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9f30a26963bc4d0aa33b95bb45032870" } new_table_name: "default.Table_6fd5c2ef64f24eb2a49b9e8f7715642b" modify_external_catalogs: true
I20251024 19:48:16.648259 1766 tablet_bootstrap.cc:654] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.641485 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_727cead5e96a4e04af23383cb5503560" } modify_external_catalogs: true
I20251024 19:48:16.650455 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6011e1b8d3634908b57b079b4dea405e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:16.651268 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4b8dc796eac74a329a4e65bbf0410b3b" } new_table_name: "default.Table_19e0e2ffa4d5450f9e3287136c9f5b32" modify_external_catalogs: true
W20251024 19:48:16.651959 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6011e1b8d3634908b57b079b4dea405e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:16.656885 1586 consensus_peers.cc:597] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:16.658310 1767 tablet_bootstrap.cc:492] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
W20251024 19:48:16.660615 1178 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet e83527f7ff5c4e248f344a84515a5d6a (table default.Table_3e341daa717746478259c784488beecf [id=c3de6937e93a48dcb88cabee59a09d32], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet e83527f7ff5c4e248f344a84515a5d6a consensus state has no leader
I20251024 19:48:16.661090 1767 ts_tablet_manager.cc:1403] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.002s
I20251024 19:48:16.665891 1775 ts_tablet_manager.cc:1929] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:16.666704 1775 log.cc:1199] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/6a7e76a3cac443ec8d2ab466075aae6b
I20251024 19:48:16.667451 1767 raft_consensus.cc:359] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.668087 1775 ts_tablet_manager.cc:1950] T 6a7e76a3cac443ec8d2ab466075aae6b P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.669883 1766 tablet_bootstrap.cc:492] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:16.670012 1766 ts_tablet_manager.cc:1403] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.004s
I20251024 19:48:16.670794 1766 raft_consensus.cc:359] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.670953 1766 raft_consensus.cc:385] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.671006 1766 raft_consensus.cc:740] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.671196 1766 consensus_queue.cc:260] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.671676 1766 ts_tablet_manager.cc:1434] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:16.671965 1766 tablet_bootstrap.cc:492] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:16.672534 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 716) took 4889 ms. Trace:
I20251024 19:48:16.673252 1767 raft_consensus.cc:385] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.673378 1767 raft_consensus.cc:740] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.673013 1775 rpcz_store.cc:276] 1024 19:48:11.783314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.783362 (+ 48us) service_pool.cc:225] Handling call
1024 19:48:16.619295 (+4835933us) ts_tablet_manager.cc:1151] Deleting tablet 6a7e76a3cac443ec8d2ab466075aae6b
1024 19:48:16.619306 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.653066 (+ 33760us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.665457 (+ 12391us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.672508 (+ 7051us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45,"tablet-delete.queue_time_us":4835724}
I20251024 19:48:16.673761 1766 tablet_bootstrap.cc:654] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.668610 1776 ts_tablet_manager.cc:1929] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251024 19:48:16.669178 1586 consensus_peers.cc:597] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:16.673856 1767 consensus_queue.cc:260] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.674440 1775 tablet_replica.cc:333] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:16.674592 1776 log.cc:1199] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/ce8c179e98944df0b510a0e8daf3920f
I20251024 19:48:16.675060 1776 ts_tablet_manager.cc:1950] T ce8c179e98944df0b510a0e8daf3920f P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:16.675546 1767 ts_tablet_manager.cc:1434] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251024 19:48:16.675575 1775 raft_consensus.cc:2243] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:16.676867 1775 raft_consensus.cc:2272] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.678720 1775 ts_tablet_manager.cc:1916] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.685290 1767 tablet_bootstrap.cc:492] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:16.688380 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 4d0e4ffcdf184c5bafd99bf721b35951 from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:16.689050 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a65fc59de5346d6a9344c5310455e53 consensus state has no leader
I20251024 19:48:16.694319 2973 raft_consensus.cc:493] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.694594 2973 raft_consensus.cc:515] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.695004 1767 tablet_bootstrap.cc:654] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.695292 2973 leader_election.cc:290] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:16.696007 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6de2c21796f4bc4904f88b654c3baec" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:16.696667 1444 leader_election.cc:343] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.696960 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6de2c21796f4bc4904f88b654c3baec" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:16.697301 1766 tablet_bootstrap.cc:492] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
W20251024 19:48:16.697413 1447 leader_election.cc:343] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.697422 1766 ts_tablet_manager.cc:1403] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.000s
I20251024 19:48:16.697511 1447 leader_election.cc:304] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:16.697539 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 744) took 4719 ms. Trace:
I20251024 19:48:16.697624 1776 rpcz_store.cc:276] 1024 19:48:11.978516 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.027239 (+ 48723us) service_pool.cc:225] Handling call
1024 19:48:16.562623 (+4535384us) ts_tablet_manager.cc:1151] Deleting tablet ce8c179e98944df0b510a0e8daf3920f
1024 19:48:16.562632 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.569347 (+ 6715us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.668512 (+ 99165us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.697520 (+ 29008us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4535165}
I20251024 19:48:16.698061 2973 raft_consensus.cc:2749] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:16.698189 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet ce8c179e98944df0b510a0e8daf3920f (table default.Table_dedd7e29ac324761b962fbd47c7c25e9 [id=870c629198ed4f53b9317b1340e75ea2]) successfully deleted
I20251024 19:48:16.698678 1766 raft_consensus.cc:359] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.698752 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 751) took 4648 ms. Trace:
I20251024 19:48:16.698907 1766 raft_consensus.cc:385] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.698994 1766 raft_consensus.cc:740] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.698902 1776 rpcz_store.cc:276] 1024 19:48:12.049881 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.049940 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:16.698560 (+4648620us) ts_tablet_manager.cc:1151] Deleting tablet 844638a6b6944d0394df3e78a8ea51d6
1024 19:48:16.698576 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.698733 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4648307}
I20251024 19:48:16.640828 1768 consensus_queue.cc:260] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.699229 1766 consensus_queue.cc:260] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:16.699533 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 844638a6b6944d0394df3e78a8ea51d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 844638a6b6944d0394df3e78a8ea51d6 already in progress: creating tablet
I20251024 19:48:16.699810 1766 ts_tablet_manager.cc:1434] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251024 19:48:16.699940 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 752) took 4649 ms. Trace:
I20251024 19:48:16.700103 1776 rpcz_store.cc:276] 1024 19:48:12.050501 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.050559 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:16.699792 (+4649233us) ts_tablet_manager.cc:1151] Deleting tablet 64d8da45c6364f1e9ddc79307904587c
1024 19:48:16.699801 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.699921 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4648920}
I20251024 19:48:16.700640 1766 tablet_bootstrap.cc:492] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:16.700611 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 64d8da45c6364f1e9ddc79307904587c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d8da45c6364f1e9ddc79307904587c already in progress: creating tablet
I20251024 19:48:16.701956 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 750) took 4652 ms. Trace:
I20251024 19:48:16.702077 1776 rpcz_store.cc:276] 1024 19:48:12.049677 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.059780 (+ 10103us) service_pool.cc:225] Handling call
1024 19:48:16.701785 (+4642005us) ts_tablet_manager.cc:1151] Deleting tablet eee675874516489f9e8dac41f70ac4e4
1024 19:48:16.701794 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.701939 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4641700}
W20251024 19:48:16.703059 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet eee675874516489f9e8dac41f70ac4e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eee675874516489f9e8dac41f70ac4e4 already in progress: creating tablet
I20251024 19:48:16.703528 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 756) took 4602 ms. Trace:
I20251024 19:48:16.703636 1766 tablet_bootstrap.cc:654] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.703685 1776 rpcz_store.cc:276] 1024 19:48:12.101527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.131768 (+ 30241us) service_pool.cc:225] Handling call
1024 19:48:16.703376 (+4571608us) ts_tablet_manager.cc:1151] Deleting tablet 373aa543e1284fbc945ca802d9eddf59
1024 19:48:16.703387 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.703509 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4571395}
W20251024 19:48:16.704290 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 373aa543e1284fbc945ca802d9eddf59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 373aa543e1284fbc945ca802d9eddf59 already in progress: creating tablet
I20251024 19:48:16.704995 2973 raft_consensus.cc:493] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.704927 1775 ts_tablet_manager.cc:1929] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.705189 2973 raft_consensus.cc:515] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.705327 1775 log.cc:1199] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/b88a6a5237d146f5b6f082ca3af25b2b
I20251024 19:48:16.705695 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 757) took 3552 ms. Trace:
I20251024 19:48:16.705790 2973 leader_election.cc:290] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:16.705778 1776 rpcz_store.cc:276] 1024 19:48:13.152813 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.152870 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:16.705540 (+3552670us) ts_tablet_manager.cc:1151] Deleting tablet 57fed826b4b64edba48c5fffaef1e031
1024 19:48:16.705549 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.705676 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3552450}
W20251024 19:48:16.706210 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 57fed826b4b64edba48c5fffaef1e031 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57fed826b4b64edba48c5fffaef1e031 already in progress: creating tablet
I20251024 19:48:16.706292 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5cbce9e4b61451ba7f458df922be759" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:16.706447 1643 raft_consensus.cc:2468] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:16.706555 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 758) took 3553 ms. Trace:
I20251024 19:48:16.706617 1776 rpcz_store.cc:276] 1024 19:48:13.153215 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.153253 (+ 38us) service_pool.cc:225] Handling call
1024 19:48:16.706421 (+3553168us) ts_tablet_manager.cc:1151] Deleting tablet 7172a93a5bec4fa1b63282ba63beec0c
1024 19:48:16.706428 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.706541 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3553031}
I20251024 19:48:16.706830 1444 leader_election.cc:304] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:16.707053 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7172a93a5bec4fa1b63282ba63beec0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7172a93a5bec4fa1b63282ba63beec0c already in progress: creating tablet
I20251024 19:48:16.707029 1775 ts_tablet_manager.cc:1950] T b88a6a5237d146f5b6f082ca3af25b2b P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.707448 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 759) took 3553 ms. Trace:
I20251024 19:48:16.707521 1776 rpcz_store.cc:276] 1024 19:48:13.153615 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.153657 (+ 42us) service_pool.cc:225] Handling call
1024 19:48:16.707284 (+3553627us) ts_tablet_manager.cc:1151] Deleting tablet be8126459ef541f39bc133f27c7d0f59
1024 19:48:16.707295 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.707432 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3553493}
W20251024 19:48:16.707968 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet be8126459ef541f39bc133f27c7d0f59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be8126459ef541f39bc133f27c7d0f59 already in progress: creating tablet
I20251024 19:48:16.708204 2973 raft_consensus.cc:2804] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:16.708285 2973 raft_consensus.cc:493] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:16.708348 2973 raft_consensus.cc:3060] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.709947 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 717) took 4884 ms. Trace:
I20251024 19:48:16.710207 1775 rpcz_store.cc:276] 1024 19:48:11.825036 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:11.827139 (+ 2103us) service_pool.cc:225] Handling call
1024 19:48:16.674187 (+4847048us) ts_tablet_manager.cc:1151] Deleting tablet b88a6a5237d146f5b6f082ca3af25b2b
1024 19:48:16.674302 (+ 115us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.691613 (+ 17311us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.704624 (+ 13011us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.709840 (+ 5216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4846853}
I20251024 19:48:16.711550 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 734) took 4667 ms. Trace:
I20251024 19:48:16.711828 1775 rpcz_store.cc:276] 1024 19:48:12.043638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.045622 (+ 1984us) service_pool.cc:225] Handling call
1024 19:48:16.711146 (+4665524us) ts_tablet_manager.cc:1151] Deleting tablet 64d8da45c6364f1e9ddc79307904587c
1024 19:48:16.711156 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.711442 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4665235}
I20251024 19:48:16.713003 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 732) took 4669 ms. Trace:
I20251024 19:48:16.713243 1775 rpcz_store.cc:276] 1024 19:48:12.043461 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.058181 (+ 14720us) service_pool.cc:225] Handling call
1024 19:48:16.712641 (+4654460us) ts_tablet_manager.cc:1151] Deleting tablet eee675874516489f9e8dac41f70ac4e4
1024 19:48:16.712652 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.712987 (+ 335us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4653586}
I20251024 19:48:16.714052 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet b88a6a5237d146f5b6f082ca3af25b2b (table default.Table_6eb10ae51f3a43c2ab15f362708fba94 [id=f3080df28ee94876bb6b790cea45a4e8]) successfully deleted
W20251024 19:48:16.714387 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 64d8da45c6364f1e9ddc79307904587c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d8da45c6364f1e9ddc79307904587c already in progress: creating tablet
W20251024 19:48:16.714740 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet eee675874516489f9e8dac41f70ac4e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eee675874516489f9e8dac41f70ac4e4 already in progress: creating tablet
I20251024 19:48:16.714818 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 733) took 4671 ms. Trace:
I20251024 19:48:16.715260 1775 rpcz_store.cc:276] 1024 19:48:12.043540 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.059249 (+ 15709us) service_pool.cc:225] Handling call
1024 19:48:16.714364 (+4655115us) ts_tablet_manager.cc:1151] Deleting tablet 844638a6b6944d0394df3e78a8ea51d6
1024 19:48:16.714377 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.714795 (+ 418us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4654527}
W20251024 19:48:16.716357 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 844638a6b6944d0394df3e78a8ea51d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 844638a6b6944d0394df3e78a8ea51d6 already in progress: creating tablet
I20251024 19:48:16.717242 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 738) took 4615 ms. Trace:
I20251024 19:48:16.717607 1775 rpcz_store.cc:276] 1024 19:48:12.102109 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.102315 (+ 206us) service_pool.cc:225] Handling call
1024 19:48:16.716854 (+4614539us) ts_tablet_manager.cc:1151] Deleting tablet 373aa543e1284fbc945ca802d9eddf59
1024 19:48:16.716954 (+ 100us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.717219 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4614306}
W20251024 19:48:16.718412 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 373aa543e1284fbc945ca802d9eddf59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 373aa543e1284fbc945ca802d9eddf59 already in progress: creating tablet
I20251024 19:48:16.718936 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 760) took 3565 ms. Trace:
I20251024 19:48:16.719076 1776 rpcz_store.cc:276] 1024 19:48:13.153890 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.153926 (+ 36us) service_pool.cc:225] Handling call
1024 19:48:16.711867 (+3557941us) ts_tablet_manager.cc:1151] Deleting tablet da7d38b63708427982591da1014aca81
1024 19:48:16.718824 (+ 6957us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.718918 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3557829}
I20251024 19:48:16.719297 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 739) took 4459 ms. Trace:
I20251024 19:48:16.719424 2973 raft_consensus.cc:515] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:16.719755 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet da7d38b63708427982591da1014aca81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da7d38b63708427982591da1014aca81 already in progress: creating tablet
I20251024 19:48:16.720075 1776 tablet_replica.cc:333] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:16.719774 1775 rpcz_store.cc:276] 1024 19:48:12.260743 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.261319 (+ 576us) service_pool.cc:225] Handling call
1024 19:48:16.718999 (+4457680us) ts_tablet_manager.cc:1151] Deleting tablet 57fed826b4b64edba48c5fffaef1e031
1024 19:48:16.719011 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.719281 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4456646}
I20251024 19:48:16.720293 1776 raft_consensus.cc:2243] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.720561 1776 raft_consensus.cc:2272] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:16.720674 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 57fed826b4b64edba48c5fffaef1e031 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57fed826b4b64edba48c5fffaef1e031 already in progress: creating tablet
I20251024 19:48:16.720872 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5cbce9e4b61451ba7f458df922be759" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:16.721065 1643 raft_consensus.cc:3060] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.721349 1776 ts_tablet_manager.cc:1916] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.721468 1766 tablet_bootstrap.cc:492] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:16.721429 1775 tablet_replica.cc:333] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:16.721611 1766 ts_tablet_manager.cc:1403] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.007s
I20251024 19:48:16.722190 1775 raft_consensus.cc:2243] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.722378 1766 raft_consensus.cc:359] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.722548 1766 raft_consensus.cc:385] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.722607 1766 raft_consensus.cc:740] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.722931 1775 raft_consensus.cc:2272] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.722843 1766 consensus_queue.cc:260] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.723712 1766 ts_tablet_manager.cc:1434] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:16.724059 1766 tablet_bootstrap.cc:492] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:16.724223 1643 raft_consensus.cc:2468] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:16.724694 1444 leader_election.cc:304] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:16.725065 2857 raft_consensus.cc:2804] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:16.724959 1775 ts_tablet_manager.cc:1916] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.725224 2857 raft_consensus.cc:697] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:16.725432 2857 consensus_queue.cc:237] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.725772 1766 tablet_bootstrap.cc:654] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.729936 2973 leader_election.cc:290] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:16.732797 1767 tablet_bootstrap.cc:492] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:16.732959 1767 ts_tablet_manager.cc:1403] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.005s
I20251024 19:48:16.734269 1767 raft_consensus.cc:359] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.734499 1767 raft_consensus.cc:385] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.734565 1767 raft_consensus.cc:740] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.734851 1767 consensus_queue.cc:260] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.740015 1767 ts_tablet_manager.cc:1434] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20251024 19:48:16.740347 1767 tablet_bootstrap.cc:492] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:16.707407 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5cbce9e4b61451ba7f458df922be759" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:16.741075 1447 leader_election.cc:343] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.730473 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5cbce9e4b61451ba7f458df922be759" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:16.742270 1447 leader_election.cc:343] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.745311 1766 tablet_bootstrap.cc:492] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:16.745452 1766 ts_tablet_manager.cc:1403] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.001s
I20251024 19:48:16.746313 1766 raft_consensus.cc:359] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.746487 1766 raft_consensus.cc:385] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.746559 1766 raft_consensus.cc:740] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.746771 1766 consensus_queue.cc:260] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.747292 1766 ts_tablet_manager.cc:1434] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:16.747583 1766 tablet_bootstrap.cc:492] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:16.749832 1766 tablet_bootstrap.cc:654] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.749922 1775 ts_tablet_manager.cc:1929] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:16.750218 1775 log.cc:1199] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/be8126459ef541f39bc133f27c7d0f59
I20251024 19:48:16.751400 1775 ts_tablet_manager.cc:1950] T be8126459ef541f39bc133f27c7d0f59 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.753712 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 741) took 4450 ms. Trace:
I20251024 19:48:16.754067 1775 rpcz_store.cc:276] 1024 19:48:12.303620 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.304026 (+ 406us) service_pool.cc:225] Handling call
1024 19:48:16.721151 (+4417125us) ts_tablet_manager.cc:1151] Deleting tablet be8126459ef541f39bc133f27c7d0f59
1024 19:48:16.721159 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.737503 (+ 16344us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.749500 (+ 11997us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.753692 (+ 4192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4416811}
I20251024 19:48:16.755244 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet be8126459ef541f39bc133f27c7d0f59 (table default.Table_0d796a82cf4b4e0f9cba35051c69293d [id=06db25030489409aa797ab8ea05dd6c5]) successfully deleted
I20251024 19:48:16.755586 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 740) took 4452 ms. Trace:
I20251024 19:48:16.756068 1768 ts_tablet_manager.cc:1434] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.156s user 0.001s sys 0.001s
I20251024 19:48:16.755983 1775 rpcz_store.cc:276] 1024 19:48:12.303466 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.304377 (+ 911us) service_pool.cc:225] Handling call
1024 19:48:16.755260 (+4450883us) ts_tablet_manager.cc:1151] Deleting tablet 7172a93a5bec4fa1b63282ba63beec0c
1024 19:48:16.755294 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.755570 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4449190}
W20251024 19:48:16.756973 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7172a93a5bec4fa1b63282ba63beec0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7172a93a5bec4fa1b63282ba63beec0c already in progress: creating tablet
I20251024 19:48:16.757817 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 742) took 4443 ms. Trace:
I20251024 19:48:16.758159 1775 rpcz_store.cc:276] 1024 19:48:12.314507 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.319597 (+ 5090us) service_pool.cc:225] Handling call
1024 19:48:16.757492 (+4437895us) ts_tablet_manager.cc:1151] Deleting tablet da7d38b63708427982591da1014aca81
1024 19:48:16.757593 (+ 101us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.757698 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4436879}
W20251024 19:48:16.758953 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet da7d38b63708427982591da1014aca81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da7d38b63708427982591da1014aca81 already in progress: creating tablet
I20251024 19:48:16.759552 1775 tablet_replica.cc:333] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:16.760298 1775 raft_consensus.cc:2243] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:16.761417 1775 raft_consensus.cc:2272] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.764609 1775 ts_tablet_manager.cc:1916] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.765882 1766 tablet_bootstrap.cc:492] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:16.766069 1766 ts_tablet_manager.cc:1403] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.001s
I20251024 19:48:16.766378 1776 ts_tablet_manager.cc:1929] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:16.766983 1766 raft_consensus.cc:359] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.767223 1766 raft_consensus.cc:385] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.767324 1766 raft_consensus.cc:740] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.767597 1766 consensus_queue.cc:260] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.768210 1766 ts_tablet_manager.cc:1434] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:16.768571 1766 tablet_bootstrap.cc:492] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:16.769244 1776 log.cc:1199] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/fb08e56db59b4db796c50db5bda93af9
I20251024 19:48:16.769640 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 688) took 4507 ms. Trace:
I20251024 19:48:16.770305 1776 ts_tablet_manager.cc:1950] T fb08e56db59b4db796c50db5bda93af9 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:16.771103 1766 tablet_bootstrap.cc:654] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.771992 1774 rpcz_store.cc:276] 1024 19:48:12.262476 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.268695 (+ 6219us) service_pool.cc:225] Handling call
1024 19:48:16.769460 (+4500765us) ts_tablet_manager.cc:1151] Deleting tablet 57fed826b4b64edba48c5fffaef1e031
1024 19:48:16.769472 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.769615 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4486252}
I20251024 19:48:16.772405 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 761) took 3618 ms. Trace:
I20251024 19:48:16.772526 1776 rpcz_store.cc:276] 1024 19:48:13.154118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.154154 (+ 36us) service_pool.cc:225] Handling call
1024 19:48:16.720003 (+3565849us) ts_tablet_manager.cc:1151] Deleting tablet fb08e56db59b4db796c50db5bda93af9
1024 19:48:16.720014 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.743738 (+ 23724us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.766276 (+ 22538us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.772385 (+ 6109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3565730}
I20251024 19:48:16.773118 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]) successfully deleted
I20251024 19:48:16.773488 1776 tablet_replica.cc:333] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:16.773757 1776 raft_consensus.cc:2243] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:16.777038 1776 raft_consensus.cc:2272] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:16.778434 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 57fed826b4b64edba48c5fffaef1e031 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57fed826b4b64edba48c5fffaef1e031 already in progress: creating tablet
I20251024 19:48:16.778705 1776 ts_tablet_manager.cc:1916] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.771466 1768 tablet_bootstrap.cc:492] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:16.779987 2920 raft_consensus.cc:493] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.780118 2920 raft_consensus.cc:515] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.780664 2920 leader_election.cc:290] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:16.781302 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4193750331874d7ba9bbe6279c3a8e37" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:16.781491 1505 raft_consensus.cc:2468] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:16.781970 1586 leader_election.cc:304] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:16.782294 2920 raft_consensus.cc:2804] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:16.782367 2920 raft_consensus.cc:493] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:16.782433 2920 raft_consensus.cc:3060] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.784226 2569 consensus_queue.cc:1048] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:16.784925 1766 tablet_bootstrap.cc:492] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:16.784875 2920 raft_consensus.cc:515] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.785050 1766 ts_tablet_manager.cc:1403] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.004s
I20251024 19:48:16.785400 2920 leader_election.cc:290] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:16.786218 1766 raft_consensus.cc:359] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.786453 1766 raft_consensus.cc:385] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.786562 1766 raft_consensus.cc:740] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.786845 1766 consensus_queue.cc:260] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.787016 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4193750331874d7ba9bbe6279c3a8e37" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:16.786859 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4193750331874d7ba9bbe6279c3a8e37" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:16.787721 1508 raft_consensus.cc:3060] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.787732 1766 ts_tablet_manager.cc:1434] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251024 19:48:16.788065 1766 tablet_bootstrap.cc:492] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:16.790145 1775 ts_tablet_manager.cc:1929] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:16.790524 1775 log.cc:1199] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/fb08e56db59b4db796c50db5bda93af9
I20251024 19:48:16.791239 1508 raft_consensus.cc:2468] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:16.791687 1775 ts_tablet_manager.cc:1950] T fb08e56db59b4db796c50db5bda93af9 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.792603 1586 leader_election.cc:304] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:16.792972 1586 leader_election.cc:343] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.793529 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4193750331874d7ba9bbe6279c3a8e37" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:16.794140 1586 leader_election.cc:343] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.794320 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 743) took 4465 ms. Trace:
I20251024 19:48:16.794538 2920 raft_consensus.cc:2804] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:16.794716 2920 raft_consensus.cc:697] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:16.794979 2920 consensus_queue.cc:237] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.794574 1775 rpcz_store.cc:276] 1024 19:48:12.329035 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.329101 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:16.759420 (+4430319us) ts_tablet_manager.cc:1151] Deleting tablet fb08e56db59b4db796c50db5bda93af9
1024 19:48:16.759428 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.777510 (+ 18082us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.789874 (+ 12364us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.794210 (+ 4336us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4429799}
I20251024 19:48:16.795783 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]) successfully deleted
I20251024 19:48:16.796134 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 689) took 4489 ms. Trace:
I20251024 19:48:16.796242 1774 rpcz_store.cc:276] 1024 19:48:12.306397 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.306455 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:16.779780 (+4473325us) ts_tablet_manager.cc:1151] Deleting tablet 7172a93a5bec4fa1b63282ba63beec0c
1024 19:48:16.779789 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.796114 (+ 16325us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4473144}
I20251024 19:48:16.796741 1775 tablet_replica.cc:333] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
W20251024 19:48:16.796737 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7172a93a5bec4fa1b63282ba63beec0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7172a93a5bec4fa1b63282ba63beec0c already in progress: creating tablet
I20251024 19:48:16.797070 1775 raft_consensus.cc:2243] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:16.797472 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 690) took 4490 ms. Trace:
I20251024 19:48:16.797755 1775 raft_consensus.cc:2272] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.798537 1775 ts_tablet_manager.cc:1916] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.803587 1696 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:16.806371 1775 ts_tablet_manager.cc:1929] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.806565 1775 log.cc:1199] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/607ae453221d422e9d07882830c02275
I20251024 19:48:16.807004 1775 ts_tablet_manager.cc:1950] T 607ae453221d422e9d07882830c02275 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.807482 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 750) took 4418 ms. Trace:
I20251024 19:48:16.807626 1775 rpcz_store.cc:276] 1024 19:48:12.388802 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.388853 (+ 51us) service_pool.cc:225] Handling call
1024 19:48:16.796670 (+4407817us) ts_tablet_manager.cc:1151] Deleting tablet 607ae453221d422e9d07882830c02275
1024 19:48:16.796682 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.802877 (+ 6195us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.806282 (+ 3405us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.807462 (+ 1180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4404843}
I20251024 19:48:16.808360 1775 tablet_replica.cc:333] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:16.808656 1775 raft_consensus.cc:2243] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.808920 1775 raft_consensus.cc:2272] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.809954 1775 ts_tablet_manager.cc:1916] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.817875 1775 ts_tablet_manager.cc:1929] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:16.818068 1775 log.cc:1199] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/1d04a9b54bd6446d8deb64292cef672d
I20251024 19:48:16.818585 1775 ts_tablet_manager.cc:1950] T 1d04a9b54bd6446d8deb64292cef672d P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.820503 1774 rpcz_store.cc:276] 1024 19:48:12.306821 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.306866 (+ 45us) service_pool.cc:225] Handling call
1024 19:48:16.797357 (+4490491us) ts_tablet_manager.cc:1151] Deleting tablet be8126459ef541f39bc133f27c7d0f59
1024 19:48:16.797367 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.797454 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4490342}
I20251024 19:48:16.821225 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 691) took 4506 ms. Trace:
I20251024 19:48:16.821422 1774 rpcz_store.cc:276] 1024 19:48:12.315229 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.315282 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:16.821080 (+4505798us) ts_tablet_manager.cc:1151] Deleting tablet da7d38b63708427982591da1014aca81
1024 19:48:16.821091 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.821204 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4505616}
I20251024 19:48:16.821967 1774 tablet_replica.cc:333] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:16.822227 1774 raft_consensus.cc:2243] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.822501 1774 raft_consensus.cc:2272] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.823468 1774 ts_tablet_manager.cc:1916] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.825647 1776 ts_tablet_manager.cc:1929] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:16.825758 1776 log.cc:1199] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/1d04a9b54bd6446d8deb64292cef672d
I20251024 19:48:16.826099 1776 ts_tablet_manager.cc:1950] T 1d04a9b54bd6446d8deb64292cef672d P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:16.826400 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 788) took 3304 ms. Trace:
I20251024 19:48:16.826491 1776 rpcz_store.cc:276] 1024 19:48:13.522414 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.522456 (+ 42us) service_pool.cc:225] Handling call
1024 19:48:16.773441 (+3250985us) ts_tablet_manager.cc:1151] Deleting tablet 1d04a9b54bd6446d8deb64292cef672d
1024 19:48:16.773451 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.808167 (+ 34716us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.825524 (+ 17357us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.826385 (+ 861us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3241824}
I20251024 19:48:16.826778 1776 tablet_replica.cc:333] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:16.826994 1776 raft_consensus.cc:2243] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.827168 1776 raft_consensus.cc:2272] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.827723 1776 ts_tablet_manager.cc:1916] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.831781 1768 tablet_bootstrap.cc:654] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.832350 1766 tablet_bootstrap.cc:654] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:16.833338 2776 log.cc:927] Time spent T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20251024 19:48:16.833720 2973 raft_consensus.cc:493] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.833853 2973 raft_consensus.cc:515] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.834503 2973 leader_election.cc:290] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:16.835129 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "844638a6b6944d0394df3e78a8ea51d6" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:16.835183 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "844638a6b6944d0394df3e78a8ea51d6" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:16.835629 1447 leader_election.cc:343] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:16.835853 1444 leader_election.cc:343] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.835971 1444 leader_election.cc:304] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:16.837800 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 607ae453221d422e9d07882830c02275 (table default.Table_e962c1d3850943a0a9e86f5882d1c518 [id=2656b4a933364e6a98f1ee5f91f0b2ba]) successfully deleted
W20251024 19:48:16.838490 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet be8126459ef541f39bc133f27c7d0f59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be8126459ef541f39bc133f27c7d0f59 already in progress: creating tablet
W20251024 19:48:16.841804 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet da7d38b63708427982591da1014aca81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da7d38b63708427982591da1014aca81 already in progress: creating tablet
I20251024 19:48:16.842696 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 1d04a9b54bd6446d8deb64292cef672d (table default.Table_147e888401aa40e5a60d97884b36cb98 [id=1acdc5d0bfb94e0faf47055b20d803b0]) successfully deleted
I20251024 19:48:16.847478 2857 raft_consensus.cc:2749] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:16.848011 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 769) took 4293 ms. Trace:
I20251024 19:48:16.848102 1775 rpcz_store.cc:276] 1024 19:48:12.554754 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.561629 (+ 6875us) service_pool.cc:225] Handling call
1024 19:48:16.808304 (+4246675us) ts_tablet_manager.cc:1151] Deleting tablet 1d04a9b54bd6446d8deb64292cef672d
1024 19:48:16.808317 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.814058 (+ 5741us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.817783 (+ 3725us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.847987 (+ 30204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4246401}
I20251024 19:48:16.848484 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 775) took 3698 ms. Trace:
I20251024 19:48:16.848549 1775 rpcz_store.cc:276] 1024 19:48:13.153481 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.153542 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:16.848352 (+3694810us) ts_tablet_manager.cc:1151] Deleting tablet 64325d89db714554866e74922d103e34
1024 19:48:16.848362 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.848468 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3694609}
I20251024 19:48:16.848806 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 777) took 3695 ms. Trace:
I20251024 19:48:16.848863 1775 rpcz_store.cc:276] 1024 19:48:13.153757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.153819 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:16.848724 (+3694905us) ts_tablet_manager.cc:1151] Deleting tablet d731403fbe0b4f92ac5899e13ea7bb19
1024 19:48:16.848732 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.848793 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3694738}
I20251024 19:48:16.849123 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 778) took 3691 ms. Trace:
I20251024 19:48:16.849180 1775 rpcz_store.cc:276] 1024 19:48:13.157758 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.157832 (+ 74us) service_pool.cc:225] Handling call
1024 19:48:16.849035 (+3691203us) ts_tablet_manager.cc:1151] Deleting tablet 769f576810ab49368eb0fc7837b7da58
1024 19:48:16.849044 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.849111 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3690965}
I20251024 19:48:16.849421 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 779) took 3682 ms. Trace:
I20251024 19:48:16.849491 1775 rpcz_store.cc:276] 1024 19:48:13.166613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.167005 (+ 392us) service_pool.cc:225] Handling call
1024 19:48:16.849340 (+3682335us) ts_tablet_manager.cc:1151] Deleting tablet fce7de9523fb4878a58aa8aef0c6b182
1024 19:48:16.849347 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.849409 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3678596}
I20251024 19:48:16.849768 1768 tablet_bootstrap.cc:492] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:16.849812 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 784) took 3580 ms. Trace:
I20251024 19:48:16.849927 1768 ts_tablet_manager.cc:1403] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.002s
I20251024 19:48:16.849879 1775 rpcz_store.cc:276] 1024 19:48:13.269723 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.275823 (+ 6100us) service_pool.cc:225] Handling call
1024 19:48:16.849688 (+3573865us) ts_tablet_manager.cc:1151] Deleting tablet b22b58404b8b441ba97ef536885a66e0
1024 19:48:16.849696 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.849795 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3573662}
I20251024 19:48:16.850103 1775 tablet_replica.cc:333] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:16.850307 1775 raft_consensus.cc:2243] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.850485 1775 raft_consensus.cc:2272] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.850829 1768 raft_consensus.cc:359] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.851020 1775 ts_tablet_manager.cc:1916] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:16.853117 1586 consensus_peers.cc:597] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:16.857991 1775 ts_tablet_manager.cc:1929] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:16.858188 1775 log.cc:1199] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/ad055dc3387b414bb55d09b9683ba17e
I20251024 19:48:16.858268 1767 tablet_bootstrap.cc:654] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.858651 1775 ts_tablet_manager.cc:1950] T ad055dc3387b414bb55d09b9683ba17e P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.859388 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 783) took 3589 ms. Trace:
I20251024 19:48:16.859524 1775 rpcz_store.cc:276] 1024 19:48:13.269511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.291340 (+ 21829us) service_pool.cc:225] Handling call
1024 19:48:16.850065 (+3558725us) ts_tablet_manager.cc:1151] Deleting tablet ad055dc3387b414bb55d09b9683ba17e
1024 19:48:16.850073 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.854959 (+ 4886us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.857900 (+ 2941us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.859372 (+ 1472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3558465}
I20251024 19:48:16.860080 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 785) took 3528 ms. Trace:
I20251024 19:48:16.860195 1775 rpcz_store.cc:276] 1024 19:48:13.331966 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.396336 (+ 64370us) service_pool.cc:225] Handling call
1024 19:48:16.859982 (+3463646us) ts_tablet_manager.cc:1151] Deleting tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4
1024 19:48:16.859991 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.860065 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3461022}
I20251024 19:48:16.860502 1775 tablet_replica.cc:333] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:16.860723 1775 raft_consensus.cc:2243] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.861028 1775 raft_consensus.cc:2272] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.861774 1775 ts_tablet_manager.cc:1916] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.865191 1776 ts_tablet_manager.cc:1929] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.866310 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 1d04a9b54bd6446d8deb64292cef672d (table default.Table_147e888401aa40e5a60d97884b36cb98 [id=1acdc5d0bfb94e0faf47055b20d803b0]) successfully deleted
W20251024 19:48:16.866899 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 64325d89db714554866e74922d103e34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64325d89db714554866e74922d103e34 already in progress: creating tablet
W20251024 19:48:16.867162 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet d731403fbe0b4f92ac5899e13ea7bb19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d731403fbe0b4f92ac5899e13ea7bb19 already in progress: creating tablet
W20251024 19:48:16.867447 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 769f576810ab49368eb0fc7837b7da58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 769f576810ab49368eb0fc7837b7da58 already in progress: creating tablet
W20251024 19:48:16.867686 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet fce7de9523fb4878a58aa8aef0c6b182 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fce7de9523fb4878a58aa8aef0c6b182 already in progress: creating tablet
W20251024 19:48:16.867995 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b22b58404b8b441ba97ef536885a66e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b22b58404b8b441ba97ef536885a66e0 already in progress: creating tablet
I20251024 19:48:16.868316 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet ad055dc3387b414bb55d09b9683ba17e (table default.Table_3e341daa717746478259c784488beecf [id=c3de6937e93a48dcb88cabee59a09d32]) successfully deleted
W20251024 19:48:16.868712 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4 already in progress: creating tablet
I20251024 19:48:16.873438 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_da7aeba16fdd4a0fa3d17ada53c2f9fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:16.874171 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da7aeba16fdd4a0fa3d17ada53c2f9fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:16.875280 1776 log.cc:1199] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/607ae453221d422e9d07882830c02275
I20251024 19:48:16.875866 1776 ts_tablet_manager.cc:1950] T 607ae453221d422e9d07882830c02275 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:16.877142 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 786) took 3355 ms. Trace:
W20251024 19:48:16.877488 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f16ef86c5eb24489a66d6cd4bfa804fa consensus state has no leader
I20251024 19:48:16.877800 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet 3b8e8dbca79f4d72a2de51f105b209e4 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:16.878261 1776 rpcz_store.cc:276] 1024 19:48:13.522161 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.522204 (+ 43us) service_pool.cc:225] Handling call
1024 19:48:16.826736 (+3304532us) ts_tablet_manager.cc:1151] Deleting tablet 607ae453221d422e9d07882830c02275
1024 19:48:16.826744 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.862091 (+ 35347us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.865111 (+ 3020us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.877122 (+ 12011us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3294949}
I20251024 19:48:16.879073 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 607ae453221d422e9d07882830c02275 (table default.Table_e962c1d3850943a0a9e86f5882d1c518 [id=2656b4a933364e6a98f1ee5f91f0b2ba]) successfully deleted
I20251024 19:48:16.879117 1776 tablet_replica.cc:333] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:16.879283 1776 raft_consensus.cc:2243] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.879837 1776 raft_consensus.cc:2272] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.880625 1776 ts_tablet_manager.cc:1916] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.881357 2857 raft_consensus.cc:493] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.881790 2857 raft_consensus.cc:515] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.883080 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f89a3bbeb9b4d57a49078729110d951" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:16.883800 1444 leader_election.cc:343] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.885146 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f89a3bbeb9b4d57a49078729110d951" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:16.885934 1447 leader_election.cc:343] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.885979 1768 raft_consensus.cc:385] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.886055 1768 raft_consensus.cc:740] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.886049 1447 leader_election.cc:304] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:16.886128 1775 ts_tablet_manager.cc:1929] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251024 19:48:16.886246 1775 log.cc:1199] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/2346c03802a245a2af8c086b626d660d
I20251024 19:48:16.886312 2973 raft_consensus.cc:2749] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:16.886303 1768 consensus_queue.cc:260] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.886693 1775 ts_tablet_manager.cc:1950] T 2346c03802a245a2af8c086b626d660d P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.888721 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 794) took 3291 ms. Trace:
I20251024 19:48:16.890833 1775 rpcz_store.cc:276] 1024 19:48:13.597405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.597776 (+ 371us) service_pool.cc:225] Handling call
1024 19:48:16.860464 (+3262688us) ts_tablet_manager.cc:1151] Deleting tablet 2346c03802a245a2af8c086b626d660d
1024 19:48:16.860474 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.870880 (+ 10406us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.886026 (+ 15146us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.888698 (+ 2672us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3262463}
I20251024 19:48:16.891548 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 2346c03802a245a2af8c086b626d660d (table default.Table_1865562320ef412799f93a15c66ade0c [id=aadca5a1b6d4494aa37f8a8cc76ad12a]) successfully deleted
I20251024 19:48:16.891964 1775 tablet_replica.cc:333] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:16.887125 2857 leader_election.cc:290] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:16.892360 1775 raft_consensus.cc:2243] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:16.892946 1775 raft_consensus.cc:2272] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.893343 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a99c55b564b946b6a76c2b981640cdcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:16.893994 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a99c55b564b946b6a76c2b981640cdcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:16.894636 1775 ts_tablet_manager.cc:1916] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.895362 2857 raft_consensus.cc:493] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.896250 2857 raft_consensus.cc:515] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.897500 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d129bdc876e425aabcf1c99997e319a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:16.898197 1447 leader_election.cc:343] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.899140 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d129bdc876e425aabcf1c99997e319a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:16.899351 1643 raft_consensus.cc:2468] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:16.899880 1444 leader_election.cc:304] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:16.900202 2973 raft_consensus.cc:2804] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:16.900279 2973 raft_consensus.cc:493] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:16.900348 2973 raft_consensus.cc:3060] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.902333 1768 ts_tablet_manager.cc:1434] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20251024 19:48:16.903497 2973 raft_consensus.cc:515] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.900975 1776 ts_tablet_manager.cc:1929] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.904608 1776 log.cc:1199] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/4243f11bf2b34b30b10766c19e2c6409
I20251024 19:48:16.904590 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d129bdc876e425aabcf1c99997e319a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:16.905130 1776 ts_tablet_manager.cc:1950] T 4243f11bf2b34b30b10766c19e2c6409 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
W20251024 19:48:16.905170 1447 leader_election.cc:343] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.905449 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 787) took 3383 ms. Trace:
I20251024 19:48:16.905535 1776 rpcz_store.cc:276] 1024 19:48:13.522290 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.522329 (+ 39us) service_pool.cc:225] Handling call
1024 19:48:16.879072 (+3356743us) ts_tablet_manager.cc:1151] Deleting tablet 4243f11bf2b34b30b10766c19e2c6409
1024 19:48:16.879081 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.885101 (+ 6020us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.900868 (+ 15767us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.905431 (+ 4563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3277518}
I20251024 19:48:16.906225 2973 leader_election.cc:290] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:16.909003 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3f24dc3f9f2e410b992b0f2a2d022513"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:16.910145 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f24dc3f9f2e410b992b0f2a2d022513 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:16.910410 1776 tablet_replica.cc:333] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:16.910601 1776 raft_consensus.cc:2243] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.910768 1776 raft_consensus.cc:2272] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.911386 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 4243f11bf2b34b30b10766c19e2c6409 (table default.Table_11634cbabbd6478484a0aed448ae9521 [id=d3b067d90c5943449b53edca7c6eb475]) successfully deleted
I20251024 19:48:16.906080 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d129bdc876e425aabcf1c99997e319a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:16.911872 1643 raft_consensus.cc:3060] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:16.912276 1776 ts_tablet_manager.cc:1916] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.914711 1643 raft_consensus.cc:2468] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:16.915576 1444 leader_election.cc:304] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:16.916649 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet ca6b8783039a4f5fbbda65c129805642 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:16.917003 2973 raft_consensus.cc:2804] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:16.917332 2973 raft_consensus.cc:697] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:16.918632 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet 0143a393711446fdbe6c798efa0c8936 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:16.919037 2973 consensus_queue.cc:237] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.916260 1768 tablet_bootstrap.cc:492] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:16.919234 1775 ts_tablet_manager.cc:1929] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.919874 1775 log.cc:1199] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/4243f11bf2b34b30b10766c19e2c6409
I20251024 19:48:16.920414 1775 ts_tablet_manager.cc:1950] T 4243f11bf2b34b30b10766c19e2c6409 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.921257 1768 tablet_bootstrap.cc:654] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.920583 2857 leader_election.cc:290] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:16.922149 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 795) took 3311 ms. Trace:
I20251024 19:48:16.925627 1775 rpcz_store.cc:276] 1024 19:48:13.610582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.610931 (+ 349us) service_pool.cc:225] Handling call
1024 19:48:16.891909 (+3280978us) ts_tablet_manager.cc:1151] Deleting tablet 4243f11bf2b34b30b10766c19e2c6409
1024 19:48:16.891920 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.906431 (+ 14511us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.919124 (+ 12693us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.922128 (+ 3004us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3280736}
I20251024 19:48:16.926405 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 4243f11bf2b34b30b10766c19e2c6409 (table default.Table_11634cbabbd6478484a0aed448ae9521 [id=d3b067d90c5943449b53edca7c6eb475]) successfully deleted
I20251024 19:48:16.926826 1775 tablet_replica.cc:333] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:16.927157 2858 raft_consensus.cc:493] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.927228 1775 raft_consensus.cc:2243] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:16.927286 2858 raft_consensus.cc:515] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.927506 1775 raft_consensus.cc:2272] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.927862 2858 leader_election.cc:290] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:16.928050 1775 ts_tablet_manager.cc:1916] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.928589 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62667092d7e244cca1020ec220ea9124" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:16.929533 1447 leader_election.cc:343] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.930644 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62667092d7e244cca1020ec220ea9124" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:16.931018 1768 tablet_bootstrap.cc:492] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
W20251024 19:48:16.931316 1444 leader_election.cc:343] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:16.931469 1444 leader_election.cc:304] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:16.931810 1768 ts_tablet_manager.cc:1403] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20251024 19:48:16.925545 1776 ts_tablet_manager.cc:1929] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.932224 1776 log.cc:1199] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/187bc968c2b049028dd25dfb0264c0aa
I20251024 19:48:16.932503 2858 raft_consensus.cc:2749] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:16.932583 1776 ts_tablet_manager.cc:1950] T 187bc968c2b049028dd25dfb0264c0aa P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:16.932889 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 794) took 3297 ms. Trace:
I20251024 19:48:16.932579 1768 raft_consensus.cc:359] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.934240 1768 raft_consensus.cc:385] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.934640 1768 raft_consensus.cc:740] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.935034 1768 consensus_queue.cc:260] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.936676 1766 tablet_bootstrap.cc:492] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:16.936869 1766 ts_tablet_manager.cc:1403] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.149s user 0.014s sys 0.023s
I20251024 19:48:16.937659 1766 raft_consensus.cc:359] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.938098 1766 raft_consensus.cc:385] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.938299 1766 raft_consensus.cc:740] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.938795 1766 consensus_queue.cc:260] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.939491 1766 ts_tablet_manager.cc:1434] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:16.940013 1766 tablet_bootstrap.cc:492] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:16.946111 1766 tablet_bootstrap.cc:654] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:16.946576 2776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Bootstrap replayed 1/1 log segments. Stats: ops{read=594 overwritten=0 applied=517 ignored=0} inserts{seen=489 ignored=0} mutations{seen=597 ignored=0} orphaned_commits=0. Pending: 77 replicates
I20251024 19:48:16.946641 1776 rpcz_store.cc:276] 1024 19:48:13.635816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.637657 (+ 1841us) service_pool.cc:225] Handling call
1024 19:48:16.910357 (+3272700us) ts_tablet_manager.cc:1151] Deleting tablet 187bc968c2b049028dd25dfb0264c0aa
1024 19:48:16.910369 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.915284 (+ 4915us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.925454 (+ 10170us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.932874 (+ 7420us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3272490}
I20251024 19:48:16.947289 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 795) took 3311 ms. Trace:
I20251024 19:48:16.947363 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 187bc968c2b049028dd25dfb0264c0aa (table default.Table_31fb0d4536b94892aeceb91307c74837 [id=2bd12943c3e24d70a4f02626d7d58f6c]) successfully deleted
I20251024 19:48:16.947413 1776 rpcz_store.cc:276] 1024 19:48:13.635890 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.637020 (+ 1130us) service_pool.cc:225] Handling call
1024 19:48:16.947155 (+3310135us) ts_tablet_manager.cc:1151] Deleting tablet 64325d89db714554866e74922d103e34
1024 19:48:16.947166 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.947273 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3303369}
W20251024 19:48:16.947918 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 64325d89db714554866e74922d103e34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64325d89db714554866e74922d103e34 already in progress: creating tablet
I20251024 19:48:16.948333 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 798) took 3311 ms. Trace:
I20251024 19:48:16.948475 1776 rpcz_store.cc:276] 1024 19:48:13.636978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.637051 (+ 73us) service_pool.cc:225] Handling call
1024 19:48:16.948218 (+3311167us) ts_tablet_manager.cc:1151] Deleting tablet 769f576810ab49368eb0fc7837b7da58
1024 19:48:16.948227 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.948317 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3230257}
W20251024 19:48:16.949275 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 769f576810ab49368eb0fc7837b7da58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 769f576810ab49368eb0fc7837b7da58 already in progress: creating tablet
I20251024 19:48:16.949678 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 797) took 3312 ms. Trace:
I20251024 19:48:16.949800 1776 rpcz_store.cc:276] 1024 19:48:13.636827 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.636879 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:16.949543 (+3312664us) ts_tablet_manager.cc:1151] Deleting tablet d731403fbe0b4f92ac5899e13ea7bb19
1024 19:48:16.949553 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.949663 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3231342}
W20251024 19:48:16.950258 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d731403fbe0b4f92ac5899e13ea7bb19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d731403fbe0b4f92ac5899e13ea7bb19 already in progress: creating tablet
I20251024 19:48:16.950579 1776 tablet_replica.cc:333] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:16.950978 1776 raft_consensus.cc:2243] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:16.952067 1776 raft_consensus.cc:2272] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:16.952698 1776 ts_tablet_manager.cc:1916] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:16.958262 1185 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet e03c488d0f264684b0897a81a04d7466 (table default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8]). Expected version 2 got 0
I20251024 19:48:16.958778 1185 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet e29abfa3bc8944aea7d1acd03e3eb897 (table default.Table_55339e02660843c3a8536a4b7a06831e [id=ba30503223dd4979824ec9c1a50a8ae2]). Expected version 1 got 0
I20251024 19:48:16.959426 1775 ts_tablet_manager.cc:1929] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:16.959535 1775 log.cc:1199] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/cef80e4d14bb4e719bceebf43041d079
I20251024 19:48:16.959908 1775 ts_tablet_manager.cc:1950] T cef80e4d14bb4e719bceebf43041d079 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:16.960538 1767 tablet_bootstrap.cc:492] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:16.960702 1767 ts_tablet_manager.cc:1403] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.220s user 0.014s sys 0.014s
I20251024 19:48:16.960656 1348 tablet_service.cc:1505] Processing CreateTablet for tablet dd9014e7efce4f4ba10d9331636c26e9 (DEFAULT_TABLE table=default.Table_b8a9d4b3797a4bd6af7122890b92c30e [id=e7ebd82dd95d4d43875158edfae8ccaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.961668 1614 tablet_service.cc:1505] Processing CreateTablet for tablet dd9014e7efce4f4ba10d9331636c26e9 (DEFAULT_TABLE table=default.Table_b8a9d4b3797a4bd6af7122890b92c30e [id=e7ebd82dd95d4d43875158edfae8ccaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.962049 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd9014e7efce4f4ba10d9331636c26e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.962888 1184 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet e03c488d0f264684b0897a81a04d7466 (table default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8]). Expected version 2 got 0
I20251024 19:48:16.963341 2923 consensus_queue.cc:1048] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:16.963476 1184 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 187925082cdd4923a6e9148bf9f270fd (table default.Table_55f8c66bd97343438051a98af12d8c2c [id=33f5ac73298543d0b5af77a27e897b79]). Expected version 1 got 0
I20251024 19:48:16.964072 1184 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet e29abfa3bc8944aea7d1acd03e3eb897 (table default.Table_55339e02660843c3a8536a4b7a06831e [id=ba30503223dd4979824ec9c1a50a8ae2]). Expected version 1 got 0
I20251024 19:48:16.964545 1184 catalog_manager.cc:5649] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:16.966373 1611 tablet_service.cc:1505] Processing CreateTablet for tablet e320bd0c6eba4d1b965e31d16234abcf (DEFAULT_TABLE table=default.Table_6c5115e46f6746acb0d4fa01a9227c6a [id=672046f5a7d34dc9a079d5c473571631]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.966373 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 40338a7939be4240b44dc54cc77ef020 (DEFAULT_TABLE table=default.Table_3219a7b4a2474607bd3df6f6d3e35642 [id=f4e3bde17c5e43b1b79d45cd302e0529]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.966769 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40338a7939be4240b44dc54cc77ef020. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.967440 1482 tablet_service.cc:1505] Processing CreateTablet for tablet dd9014e7efce4f4ba10d9331636c26e9 (DEFAULT_TABLE table=default.Table_b8a9d4b3797a4bd6af7122890b92c30e [id=e7ebd82dd95d4d43875158edfae8ccaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.967623 1479 tablet_service.cc:1505] Processing CreateTablet for tablet e320bd0c6eba4d1b965e31d16234abcf (DEFAULT_TABLE table=default.Table_6c5115e46f6746acb0d4fa01a9227c6a [id=672046f5a7d34dc9a079d5c473571631]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.967831 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd9014e7efce4f4ba10d9331636c26e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.968243 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e320bd0c6eba4d1b965e31d16234abcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.968585 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 40338a7939be4240b44dc54cc77ef020 (DEFAULT_TABLE table=default.Table_3219a7b4a2474607bd3df6f6d3e35642 [id=f4e3bde17c5e43b1b79d45cd302e0529]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.968586 1345 tablet_service.cc:1505] Processing CreateTablet for tablet e320bd0c6eba4d1b965e31d16234abcf (DEFAULT_TABLE table=default.Table_6c5115e46f6746acb0d4fa01a9227c6a [id=672046f5a7d34dc9a079d5c473571631]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.969795 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e320bd0c6eba4d1b965e31d16234abcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.973524 1187 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_933a6697624149369ab3ec28462b466a [id=93a61862707b4a7ba8a0ff349704dc2b] alter complete (version 1)
W20251024 19:48:16.974643 1183 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.252.193:37629 (request call id 158) took 1041 ms (client timeout 1000 ms). Trace:
W20251024 19:48:16.975062 1187 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.252.194:36397 (request call id 123) took 1363 ms (client timeout 1000 ms). Trace:
W20251024 19:48:16.974787 1183 rpcz_store.cc:269] 1024 19:48:15.933074 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.312808 (+379734us) service_pool.cc:225] Handling call
1024 19:48:16.974573 (+661765us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":8}
I20251024 19:48:16.975780 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b89a90bb4ca447fbaf34fe39001c261 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:16.976030 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fd8aa1d9aca429dbfe46f472ebbf53b) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:16.976210 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet fdfa5eea3b674a3389a156d1c72204df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:16.976598 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet ce8c179e98944df0b510a0e8daf3920f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:16.975237 1187 rpcz_store.cc:269] 1024 19:48:15.611383 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.638636 (+ 27253us) service_pool.cc:225] Handling call
1024 19:48:15.642582 (+ 3946us) spinlock_profiling.cc:243] Waited 2.74 ms on lock 0x616000041254. stack: 00007f847ed6f9f6 00007f847ed6f7b0 00007f847dd4b3ad 0000000000983f04 0000000000983ecd 00007f848e06d6bd 00007f848e06d7b3 00007f848ac38441 00007f84928ee9dd 00007f8492a75aa8 00007f848958eca6 00007f848958e9d5 00007f8485d0b27e 00007f8485d0a15e 00007f8485d10395 00007f8485d10cbf
1024 19:48:16.974974 (+1332392us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1024 19:48:15.731789 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1024 19:48:15.731895 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
1024 19:48:15.731902 (+ 7us) write_op.cc:435] Acquired schema lock
1024 19:48:15.731907 (+ 5us) tablet.cc:660] Decoding operations
1024 19:48:15.731990 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
1024 19:48:15.732001 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1024 19:48:15.732004 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
1024 19:48:15.732052 (+ 48us) tablet.cc:699] Row locks acquired
1024 19:48:15.732054 (+ 2us) write_op.cc:260] PREPARE: finished
1024 19:48:15.732088 (+ 34us) write_op.cc:270] Start()
1024 19:48:15.732117 (+ 29us) write_op.cc:276] Timestamp: P: 1761335295731100 usec, L: 0
1024 19:48:15.732120 (+ 3us) op_driver.cc:348] REPLICATION: starting
1024 19:48:15.732280 (+ 160us) log.cc:844] Serialized 356 byte log entry
1024 19:48:15.739144 (+ 6864us) op_driver.cc:464] REPLICATION: finished
1024 19:48:15.740766 (+ 1622us) write_op.cc:301] APPLY: starting
1024 19:48:15.740784 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1024 19:48:15.740818 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
1024 19:48:15.740844 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
1024 19:48:15.740902 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
1024 19:48:15.740934 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1024 19:48:15.740941 (+ 7us) write_op.cc:312] APPLY: finished
1024 19:48:15.741045 (+ 104us) log.cc:844] Serialized 23 byte log entry
1024 19:48:15.741103 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:15.741121 (+ 18us) write_op.cc:454] Released schema lock
1024 19:48:15.741127 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1024 19:48:15.742355 (+ 1228us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:15.742361 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
1024 19:48:16.482954 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1024 19:48:16.483048 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
1024 19:48:16.483054 (+ 6us) write_op.cc:435] Acquired schema lock
1024 19:48:16.483056 (+ 2us) tablet.cc:660] Decoding operations
1024 19:48:16.483137 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
1024 19:48:16.483146 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1024 19:48:16.483151 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1024 19:48:16.483219 (+ 68us) tablet.cc:699] Row locks acquired
1024 19:48:16.483221 (+ 2us) write_op.cc:260] PREPARE: finished
1024 19:48:16.483253 (+ 32us) write_op.cc:270] Start()
1024 19:48:16.483281 (+ 28us) write_op.cc:276] Timestamp: P: 1761335296483206 usec, L: 0
1024 19:48:16.483284 (+ 3us) op_driver.cc:348] REPLICATION: starting
1024 19:48:16.483430 (+ 146us) log.cc:844] Serialized 356 byte log entry
1024 19:48:16.630100 (+146670us) op_driver.cc:464] REPLICATION: finished
1024 19:48:16.632787 (+ 2687us) write_op.cc:301] APPLY: starting
1024 19:48:16.632799 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1024 19:48:16.632829 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
1024 19:48:16.632872 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
1024 19:48:16.632908 (+ 36us) tablet.cc:1381] finished ApplyRowOperation cycle
1024 19:48:16.632932 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1024 19:48:16.632938 (+ 6us) write_op.cc:312] APPLY: finished
1024 19:48:16.633038 (+ 100us) log.cc:844] Serialized 23 byte log entry
1024 19:48:16.633094 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:16.633104 (+ 10us) write_op.cc:454] Released schema lock
1024 19:48:16.633108 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1024 19:48:16.653883 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1024 19:48:16.653969 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
1024 19:48:16.653975 (+ 6us) write_op.cc:435] Acquired schema lock
1024 19:48:16.653978 (+ 3us) tablet.cc:660] Decoding operations
1024 19:48:16.654069 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
1024 19:48:16.654080 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1024 19:48:16.654086 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1024 19:48:16.654147 (+ 61us) tablet.cc:699] Row locks acquired
1024 19:48:16.654150 (+ 3us) write_op.cc:260] PREPARE: finished
1024 19:48:16.654187 (+ 37us) write_op.cc:270] Start()
1024 19:48:16.654218 (+ 31us) write_op.cc:276] Timestamp: P: 1761335296654162 usec, L: 0
1024 19:48:16.654222 (+ 4us) op_driver.cc:348] REPLICATION: starting
1024 19:48:16.654381 (+ 159us) log.cc:844] Serialized 356 byte log entry
1024 19:48:16.686154 (+ 31773us) op_driver.cc:464] REPLICATION: finished
1024 19:48:16.686706 (+ 552us) write_op.cc:301] APPLY: starting
1024 19:48:16.686731 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
1024 19:48:16.686761 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
1024 19:48:16.686783 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
1024 19:48:16.686823 (+ 40us) tablet.cc:1381] finished ApplyRowOperation cycle
1024 19:48:16.686850 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1024 19:48:16.686857 (+ 7us) write_op.cc:312] APPLY: finished
1024 19:48:16.686912 (+ 55us) log.cc:844] Serialized 23 byte log entry
1024 19:48:16.686956 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:16.686971 (+ 15us) write_op.cc:454] Released schema lock
1024 19:48:16.686978 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1024 19:48:16.687047 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:16.687051 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":12,"spinlock_wait_cycles":6290816,"child_traces":[["op",{"apply.queue_time_us":1557,"apply.run_cpu_time_us":431,"apply.run_wall_time_us":432,"num_ops":1,"prepare.queue_time_us":56122,"prepare.run_cpu_time_us":615,"prepare.run_wall_time_us":10553,"raft.queue_time_us":41,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":219,"replication_time_us":6995}],["op",{"apply.queue_time_us":2614,"apply.run_cpu_time_us":382,"apply.run_wall_time_us":447,"num_ops":1,"prepare.queue_time_us":519843,"prepare.run_cpu_time_us":543,"prepare.run_wall_time_us":819,"replication_time_us":146782}],["op",{"apply.queue_time_us":512,"apply.run_cpu_time_us":331,"apply.run_wall_time_us":330,"num_ops":1,"prepare.queue_time_us":13115,"prepare.run_cpu_time_us":598,"prepare.run_wall_time_us":2005,"raft.queue_time_us":316,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":31910}]]}
I20251024 19:48:16.977165 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet 95f62010a24f432c84c89345f9b02b2f from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:16.977682 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet f83b9290f00145a780318a43249eec12 with delete_type TABLET_DATA_DELETED (replaced by tablet 41176885447e42409ee4363bb4d903f2) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:16.977744 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet c82f60819e7a442bbc0cc04659e14e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:16.973214 1189 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_ea0d3589c13a4ebeb950ddc737a9948b [id=350b37bf42194fcfbb660bffe83e47c4] alter complete (version 4)
I20251024 19:48:16.978325 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 7352d84226db467dad31ae5941842b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:16.980262 1189 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.252.195:44995 (request call id 139) took 1306 ms (client timeout 1000 ms). Trace:
I20251024 19:48:16.980166 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f052d9f6feec4b35be9654644bbb59c9" } new_table_name: "default.Table_32fae81421354505aa0fe8cd26358df2" modify_external_catalogs: true
W20251024 19:48:16.980365 1189 rpcz_store.cc:269] 1024 19:48:15.674105 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.683179 (+ 9074us) service_pool.cc:225] Handling call
1024 19:48:16.980200 (+1297021us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1024 19:48:16.483812 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1024 19:48:16.483898 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
1024 19:48:16.483903 (+ 5us) write_op.cc:435] Acquired schema lock
1024 19:48:16.483920 (+ 17us) tablet.cc:660] Decoding operations
1024 19:48:16.483995 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
1024 19:48:16.484004 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1024 19:48:16.484008 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1024 19:48:16.484052 (+ 44us) tablet.cc:699] Row locks acquired
1024 19:48:16.484054 (+ 2us) write_op.cc:260] PREPARE: finished
1024 19:48:16.484080 (+ 26us) write_op.cc:270] Start()
1024 19:48:16.484103 (+ 23us) write_op.cc:276] Timestamp: P: 1761335296484036 usec, L: 0
1024 19:48:16.484106 (+ 3us) op_driver.cc:348] REPLICATION: starting
1024 19:48:16.484226 (+ 120us) log.cc:844] Serialized 533 byte log entry
1024 19:48:16.635338 (+151112us) op_driver.cc:464] REPLICATION: finished
1024 19:48:16.635497 (+ 159us) write_op.cc:301] APPLY: starting
1024 19:48:16.635509 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1024 19:48:16.635537 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
1024 19:48:16.635564 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1024 19:48:16.635603 (+ 39us) tablet.cc:1381] finished ApplyRowOperation cycle
1024 19:48:16.635628 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1024 19:48:16.635634 (+ 6us) write_op.cc:312] APPLY: finished
1024 19:48:16.635689 (+ 55us) log.cc:844] Serialized 23 byte log entry
1024 19:48:16.635736 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:16.635748 (+ 12us) write_op.cc:454] Released schema lock
1024 19:48:16.635753 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1024 19:48:16.635820 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:16.635824 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1024 19:48:16.638090 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1024 19:48:16.638247 (+ 157us) write_op.cc:432] Acquiring schema lock in shared mode
1024 19:48:16.638251 (+ 4us) write_op.cc:435] Acquired schema lock
1024 19:48:16.638253 (+ 2us) tablet.cc:660] Decoding operations
1024 19:48:16.638344 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
1024 19:48:16.638354 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1024 19:48:16.638359 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1024 19:48:16.638439 (+ 80us) tablet.cc:699] Row locks acquired
1024 19:48:16.638442 (+ 3us) write_op.cc:260] PREPARE: finished
1024 19:48:16.638478 (+ 36us) write_op.cc:270] Start()
1024 19:48:16.638506 (+ 28us) write_op.cc:276] Timestamp: P: 1761335296638453 usec, L: 0
1024 19:48:16.638511 (+ 5us) op_driver.cc:348] REPLICATION: starting
1024 19:48:16.649772 (+ 11261us) log.cc:844] Serialized 356 byte log entry
1024 19:48:16.653777 (+ 4005us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x616000041254. stack: 00007f847ed6f9f6 00007f847ed6f7b0 00007f847dd4b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f848ac0a862 00007f848bfed2a5 00007f848bfebdc9 00007f848bff41f0 00007f848bff3ffd 0000000000a8867c 00007f847edb5d7f 00007f847edb904b 00007f847edb8ec8 0000000000a8867c
1024 19:48:16.686056 (+ 32279us) op_driver.cc:464] REPLICATION: finished
1024 19:48:16.686194 (+ 138us) write_op.cc:301] APPLY: starting
1024 19:48:16.686213 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1024 19:48:16.686264 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
1024 19:48:16.686268 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
1024 19:48:16.686334 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
1024 19:48:16.686369 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1024 19:48:16.686375 (+ 6us) write_op.cc:312] APPLY: finished
1024 19:48:16.686460 (+ 85us) log.cc:844] Serialized 23 byte log entry
1024 19:48:16.686526 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:16.686544 (+ 18us) write_op.cc:454] Released schema lock
1024 19:48:16.686551 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1024 19:48:16.686625 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:16.686629 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4601,"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":116,"apply.run_cpu_time_us":316,"apply.run_wall_time_us":316,"num_ops":1,"prepare.queue_time_us":520175,"prepare.run_cpu_time_us":492,"prepare.run_wall_time_us":723,"replication_time_us":151205}],["op",{"apply.queue_time_us":71,"apply.run_cpu_time_us":425,"apply.run_wall_time_us":425,"num_ops":1,"prepare.queue_time_us":665,"prepare.run_cpu_time_us":1132,"prepare.run_wall_time_us":15762,"raft.queue_time_us":975,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":232,"replication_time_us":47503,"spinlock_wait_cycles":25474432}]]}
I20251024 19:48:16.982210 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 27b32126d04e480f8db7b2b20aba5873 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:47:59 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:16.982443 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet cef80e4d14bb4e719bceebf43041d079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:16.982487 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet f51c56337ebb450aaf62dcc81ef63ae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:16.985193 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet b1fa439afe5a48a79557019687319af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:16.981489 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4f873edb1c0d444a88c3485b0aaa7623" } new_table_name: "default.Table_534425b79d834682916f62eb182e6869" modify_external_catalogs: true
I20251024 19:48:16.987950 1767 raft_consensus.cc:359] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.988775 1767 raft_consensus.cc:385] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:16.988750 2936 raft_consensus.cc:493] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:16.988905 2936 raft_consensus.cc:515] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:16.989157 1767 raft_consensus.cc:740] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:16.989404 1767 consensus_queue.cc:260] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:16.989555 2936 leader_election.cc:290] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:16.991577 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 40338a7939be4240b44dc54cc77ef020 (DEFAULT_TABLE table=default.Table_3219a7b4a2474607bd3df6f6d3e35642 [id=f4e3bde17c5e43b1b79d45cd302e0529]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.991606 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 09bf5103db0a4ef7912a0f8e6e017d3a (DEFAULT_TABLE table=default.Table_b809079c0a9a4540814e848853c374cf [id=73a53480761f42b2a57da67cfeaa376f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.991983 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1fa439afe5a48a79557019687319af6" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:16.992895 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 09bf5103db0a4ef7912a0f8e6e017d3a (DEFAULT_TABLE table=default.Table_b809079c0a9a4540814e848853c374cf [id=73a53480761f42b2a57da67cfeaa376f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:16.993362 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09bf5103db0a4ef7912a0f8e6e017d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.995995 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3b8e8dbca79f4d72a2de51f105b209e4
I20251024 19:48:16.996701 1188 catalog_manager.cc:6719] ReplaceTablet: tablet 3b8e8dbca79f4d72a2de51f105b209e4 deleted and replaced by tablet fc50e9fbbefb43fda123ace9d1f32cf8
I20251024 19:48:16.997267 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9ff460fc04ce46c0a4ae4163efee611b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:16.998286 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 0143a393711446fdbe6c798efa0c8936
I20251024 19:48:16.999060 1191 catalog_manager.cc:6719] ReplaceTablet: tablet 0143a393711446fdbe6c798efa0c8936 deleted and replaced by tablet e879d4c72cda4bc781a2ebdf8cc3a179
I20251024 19:48:16.999579 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6233280a74da4d1981019506fd1be2f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:17.000180 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4f00719c20934655bc744446194f7401
W20251024 19:48:17.000231 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6233280a74da4d1981019506fd1be2f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.000928 1186 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51416 (request call id 1040) took 1102 ms. Trace:
I20251024 19:48:17.001097 1186 rpcz_store.cc:276] 1024 19:48:15.898757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.575654 (+676897us) service_pool.cc:225] Handling call
1024 19:48:16.641155 (+ 65501us) spinlock_profiling.cc:243] Waited 65.2 ms on lock 0x616000041254. stack: 00007f847ed6f9f6 00007f847ed6f7b0 00007f847dd4b3ad 0000000000983f04 0000000000983ecd 00007f848e06d6bd 00007f848e06d7b3 00007f848ac38441 00007f84928ee9dd 00007f8492a7c256 00007f8489593486 00007f84895931b5 00007f8485d0b27e 00007f8485d0a15e 00007f8485d10395 00007f8485d10cbf
1024 19:48:16.649925 (+ 8770us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1024 19:48:16.650067 (+ 142us) catalog_manager.cc:2815] Modifying in-memory table state
1024 19:48:16.650148 (+ 81us) catalog_manager.cc:2824] Locking tablets
1024 19:48:16.955221 (+305073us) catalog_manager.cc:2838] Removing table and tablets from system table
1024 19:48:16.999934 (+ 44713us) catalog_manager.cc:2856] Removing table from by-name map
1024 19:48:17.000005 (+ 71us) catalog_manager.cc:2872] Committing in-memory state
1024 19:48:17.000043 (+ 38us) catalog_manager.cc:2876] Aborting table tasks
1024 19:48:17.000755 (+ 712us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1024 19:48:16.956417 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1024 19:48:16.956517 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
1024 19:48:16.956521 (+ 4us) write_op.cc:435] Acquired schema lock
1024 19:48:16.956524 (+ 3us) tablet.cc:660] Decoding operations
1024 19:48:16.956601 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
1024 19:48:16.956609 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1024 19:48:16.956613 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1024 19:48:16.956709 (+ 96us) tablet.cc:699] Row locks acquired
1024 19:48:16.956711 (+ 2us) write_op.cc:260] PREPARE: finished
1024 19:48:16.956738 (+ 27us) write_op.cc:270] Start()
1024 19:48:16.956759 (+ 21us) write_op.cc:276] Timestamp: P: 1761335296956690 usec, L: 0
1024 19:48:16.956762 (+ 3us) op_driver.cc:348] REPLICATION: starting
1024 19:48:16.956885 (+ 123us) log.cc:844] Serialized 762 byte log entry
1024 19:48:16.993970 (+ 37085us) op_driver.cc:464] REPLICATION: finished
1024 19:48:16.994715 (+ 745us) write_op.cc:301] APPLY: starting
1024 19:48:16.994725 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1024 19:48:16.994758 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
1024 19:48:16.994778 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
1024 19:48:16.994834 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
1024 19:48:16.994861 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1024 19:48:16.994867 (+ 6us) write_op.cc:312] APPLY: finished
1024 19:48:16.994923 (+ 56us) log.cc:844] Serialized 29 byte log entry
1024 19:48:16.994968 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:16.994983 (+ 15us) write_op.cc:454] Released schema lock
1024 19:48:16.994988 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1024 19:48:16.995053 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:16.995057 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":150023552,"child_traces":[["op",{"apply.queue_time_us":712,"apply.run_cpu_time_us":331,"apply.run_wall_time_us":331,"num_ops":2,"prepare.queue_time_us":389,"prepare.run_cpu_time_us":533,"prepare.run_wall_time_us":533,"replication_time_us":37191}]]}
I20251024 19:48:17.002576 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4d0e4ffcdf184c5bafd99bf721b35951
I20251024 19:48:17.003309 1192 catalog_manager.cc:6719] ReplaceTablet: tablet 4d0e4ffcdf184c5bafd99bf721b35951 deleted and replaced by tablet 03d8ab62b47f474198841ea14a3b7b00
I20251024 19:48:17.003798 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9bc761287e4a44ac958794e2ba5b7b79" } modify_external_catalogs: true
I20251024 19:48:17.004033 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9bc761287e4a44ac958794e2ba5b7b79" } modify_external_catalogs: true
I20251024 19:48:17.006038 1767 ts_tablet_manager.cc:1434] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20251024 19:48:17.008819 1768 ts_tablet_manager.cc:1434] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.077s user 0.001s sys 0.001s
W20251024 19:48:17.011677 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ff460fc04ce46c0a4ae4163efee611b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.013799 2776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: Bootstrap complete.
I20251024 19:48:17.006177 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 0143a393711446fdbe6c798efa0c8936 with delete_type TABLET_DATA_DELETED (replaced by tablet e879d4c72cda4bc781a2ebdf8cc3a179) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.006423 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8e8dbca79f4d72a2de51f105b209e4 with delete_type TABLET_DATA_DELETED (replaced by tablet fc50e9fbbefb43fda123ace9d1f32cf8) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.006625 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 2944733a2920443e991f84379893370c (DEFAULT_TABLE table=default.Table_a9ab11ad10c74cd1be885016fa225113 [id=2739509ed3ea44358385e19878c3eb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.015491 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2944733a2920443e991f84379893370c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.015563 1767 tablet_bootstrap.cc:492] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:17.017107 1767 tablet_bootstrap.cc:654] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.010660 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8e8dbca79f4d72a2de51f105b209e4 with delete_type TABLET_DATA_DELETED (replaced by tablet fc50e9fbbefb43fda123ace9d1f32cf8) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.011134 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d0e4ffcdf184c5bafd99bf721b35951 with delete_type TABLET_DATA_DELETED (replaced by tablet 03d8ab62b47f474198841ea14a3b7b00) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.011215 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet 0143a393711446fdbe6c798efa0c8936 with delete_type TABLET_DATA_DELETED (replaced by tablet e879d4c72cda4bc781a2ebdf8cc3a179) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.011329 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f00719c20934655bc744446194f7401 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:16 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.011279 1339 tablet_service.cc:1505] Processing CreateTablet for tablet a2cc3f3d02324c1e867b8fc159ad6d20 (DEFAULT_TABLE table=default.Table_2461bf28f0bd44ac8e86c5bd843a6718 [id=690c18ceb01b4fe2be369ea7ddc08c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.011154 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 2944733a2920443e991f84379893370c (DEFAULT_TABLE table=default.Table_a9ab11ad10c74cd1be885016fa225113 [id=2739509ed3ea44358385e19878c3eb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.018769 1372 raft_consensus.cc:3060] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.019447 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_15b382636dbe430c991bff2369719fc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:17.020391 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15b382636dbe430c991bff2369719fc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.006822 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f00719c20934655bc744446194f7401 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:16 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.022075 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ca6b8783039a4f5fbbda65c129805642
I20251024 19:48:17.022672 1190 catalog_manager.cc:6719] ReplaceTablet: tablet ca6b8783039a4f5fbbda65c129805642 deleted and replaced by tablet 2d8b8a1cff3540f7b7f13712425d534e
I20251024 19:48:17.023089 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f000460d66bd46a2bcae726de4f6fb1f" } modify_external_catalogs: true
I20251024 19:48:17.023295 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f000460d66bd46a2bcae726de4f6fb1f" } modify_external_catalogs: true
I20251024 19:48:17.006932 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d0e4ffcdf184c5bafd99bf721b35951 with delete_type TABLET_DATA_DELETED (replaced by tablet 03d8ab62b47f474198841ea14a3b7b00) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:17.007745 1586 consensus_peers.cc:597] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:17.012058 1617 tablet_service.cc:1505] Processing CreateTablet for tablet a2cc3f3d02324c1e867b8fc159ad6d20 (DEFAULT_TABLE table=default.Table_2461bf28f0bd44ac8e86c5bd843a6718 [id=690c18ceb01b4fe2be369ea7ddc08c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.024312 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2cc3f3d02324c1e867b8fc159ad6d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.024590 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 1dadb9331a5a45049c728783aa70b46d (DEFAULT_TABLE table=default.Table_9cbca7d5292d44c49d2d550aa9e18e92 [id=9d849696504d4ebeadd33fee5afd2cca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.024920 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dadb9331a5a45049c728783aa70b46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.025094 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 1dadb9331a5a45049c728783aa70b46d (DEFAULT_TABLE table=default.Table_9cbca7d5292d44c49d2d550aa9e18e92 [id=9d849696504d4ebeadd33fee5afd2cca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.029596 2936 raft_consensus.cc:493] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.029718 2936 raft_consensus.cc:515] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.030341 2936 leader_election.cc:290] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:17.032415 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet ca6b8783039a4f5fbbda65c129805642 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d8b8a1cff3540f7b7f13712425d534e) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.032599 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 8ff0ec53ae71461baf2b8f9ec065530f (DEFAULT_TABLE table=default.Table_09c35871ae6345f8bd692a8e416323e5 [id=64f9a8f7a56c4a32a12059f106d879c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.032929 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ff0ec53ae71461baf2b8f9ec065530f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:16.990231 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1fa439afe5a48a79557019687319af6" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:17.037338 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40338a7939be4240b44dc54cc77ef020. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.037636 1308 leader_election.cc:304] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.038084 2936 raft_consensus.cc:2749] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.039041 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 5688b62d37d6453987bb4088c849986b (DEFAULT_TABLE table=default.Table_809ec8b3a0fc4fa08457269a52eae680 [id=85703abd14a64b90863866650f7a6cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.039575 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5688b62d37d6453987bb4088c849986b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.044625 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 8ff0ec53ae71461baf2b8f9ec065530f (DEFAULT_TABLE table=default.Table_09c35871ae6345f8bd692a8e416323e5 [id=64f9a8f7a56c4a32a12059f106d879c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.044907 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet ca6b8783039a4f5fbbda65c129805642 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d8b8a1cff3540f7b7f13712425d534e) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.045367 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 5688b62d37d6453987bb4088c849986b (DEFAULT_TABLE table=default.Table_809ec8b3a0fc4fa08457269a52eae680 [id=85703abd14a64b90863866650f7a6cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.046460 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd9014e7efce4f4ba10d9331636c26e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.048909 1480 tablet_service.cc:1505] Processing CreateTablet for tablet 5688b62d37d6453987bb4088c849986b (DEFAULT_TABLE table=default.Table_809ec8b3a0fc4fa08457269a52eae680 [id=85703abd14a64b90863866650f7a6cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.049247 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5688b62d37d6453987bb4088c849986b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.049804 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40338a7939be4240b44dc54cc77ef020. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.014540 1774 ts_tablet_manager.cc:1929] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:17.052301 1774 log.cc:1199] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/fb08e56db59b4db796c50db5bda93af9
I20251024 19:48:17.052704 1774 ts_tablet_manager.cc:1950] T fb08e56db59b4db796c50db5bda93af9 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:17.054256 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e320bd0c6eba4d1b965e31d16234abcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.056460 1341 tablet_service.cc:1505] Processing CreateTablet for tablet b1f33d405c584efaafd992bf9c3f2f9c (DEFAULT_TABLE table=default.Table_6ab1b43d9b5e4537bd38fdb50f100fc1 [id=61baedf0596642c4b4495f178cea6f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.056826 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f33d405c584efaafd992bf9c3f2f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.057806 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09bf5103db0a4ef7912a0f8e6e017d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.057963 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2944733a2920443e991f84379893370c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.061152 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dadb9331a5a45049c728783aa70b46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.062325 1611 tablet_service.cc:1505] Processing CreateTablet for tablet b1f33d405c584efaafd992bf9c3f2f9c (DEFAULT_TABLE table=default.Table_6ab1b43d9b5e4537bd38fdb50f100fc1 [id=61baedf0596642c4b4495f178cea6f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.062378 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 76837fecac49493ebd5d24f192b0f693 (DEFAULT_TABLE table=default.Table_4865fd997563401387e8f190fb26e86a [id=833676d1b4444dbc81ede41ca2869e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.062692 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76837fecac49493ebd5d24f192b0f693. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.062707 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 76837fecac49493ebd5d24f192b0f693 (DEFAULT_TABLE table=default.Table_4865fd997563401387e8f190fb26e86a [id=833676d1b4444dbc81ede41ca2869e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.065987 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ff0ec53ae71461baf2b8f9ec065530f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.069846 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5688b62d37d6453987bb4088c849986b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.071020 1480 tablet_service.cc:1505] Processing CreateTablet for tablet b1f33d405c584efaafd992bf9c3f2f9c (DEFAULT_TABLE table=default.Table_6ab1b43d9b5e4537bd38fdb50f100fc1 [id=61baedf0596642c4b4495f178cea6f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.071377 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f33d405c584efaafd992bf9c3f2f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.075384 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 76837fecac49493ebd5d24f192b0f693 (DEFAULT_TABLE table=default.Table_4865fd997563401387e8f190fb26e86a [id=833676d1b4444dbc81ede41ca2869e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.075748 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76837fecac49493ebd5d24f192b0f693. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.075964 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2cc3f3d02324c1e867b8fc159ad6d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.055083 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 692) took 4722 ms. Trace:
I20251024 19:48:17.076115 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c77f6bbf7b145389ca6cf74ddd3aa3f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:17.076135 1774 rpcz_store.cc:276] 1024 19:48:12.332292 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.334693 (+ 2401us) service_pool.cc:225] Handling call
1024 19:48:16.821919 (+4487226us) ts_tablet_manager.cc:1151] Deleting tablet fb08e56db59b4db796c50db5bda93af9
1024 19:48:16.821929 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.880801 (+ 58872us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.014449 (+133648us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.055058 (+ 40609us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4487018}
I20251024 19:48:17.076634 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet fb08e56db59b4db796c50db5bda93af9 (table default.Table_08dbb5c3a0924b1491e302cef50518c4 [id=10bb93537e8344b2bf08a46e2f1109a9]) successfully deleted
I20251024 19:48:17.077348 1774 tablet_replica.cc:333] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:17.077541 1774 raft_consensus.cc:2243] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:16.998684 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 796) took 3387 ms. Trace:
I20251024 19:48:17.014703 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 09bf5103db0a4ef7912a0f8e6e017d3a (DEFAULT_TABLE table=default.Table_b809079c0a9a4540814e848853c374cf [id=73a53480761f42b2a57da67cfeaa376f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.077723 1774 raft_consensus.cc:2272] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.077981 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09bf5103db0a4ef7912a0f8e6e017d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.078245 1774 ts_tablet_manager.cc:1916] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.015029 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8e8dbca79f4d72a2de51f105b209e4 with delete_type TABLET_DATA_DELETED (replaced by tablet fc50e9fbbefb43fda123ace9d1f32cf8) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.015280 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 0143a393711446fdbe6c798efa0c8936 with delete_type TABLET_DATA_DELETED (replaced by tablet e879d4c72cda4bc781a2ebdf8cc3a179) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.015667 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f00719c20934655bc744446194f7401 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:16 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.016132 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 2944733a2920443e991f84379893370c (DEFAULT_TABLE table=default.Table_a9ab11ad10c74cd1be885016fa225113 [id=2739509ed3ea44358385e19878c3eb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.023497 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d0e4ffcdf184c5bafd99bf721b35951 with delete_type TABLET_DATA_DELETED (replaced by tablet 03d8ab62b47f474198841ea14a3b7b00) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.030747 1475 tablet_service.cc:1505] Processing CreateTablet for tablet a2cc3f3d02324c1e867b8fc159ad6d20 (DEFAULT_TABLE table=default.Table_2461bf28f0bd44ac8e86c5bd843a6718 [id=690c18ceb01b4fe2be369ea7ddc08c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.031333 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c77f6bbf7b145389ca6cf74ddd3aa3f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:17.081990 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2944733a2920443e991f84379893370c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.082296 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2cc3f3d02324c1e867b8fc159ad6d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.083130 1308 leader_election.cc:304] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.083624 2936 raft_consensus.cc:2749] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.033730 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet ca6b8783039a4f5fbbda65c129805642 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d8b8a1cff3540f7b7f13712425d534e) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.033782 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 1dadb9331a5a45049c728783aa70b46d (DEFAULT_TABLE table=default.Table_9cbca7d5292d44c49d2d550aa9e18e92 [id=9d849696504d4ebeadd33fee5afd2cca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.086099 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dadb9331a5a45049c728783aa70b46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.034332 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 8ff0ec53ae71461baf2b8f9ec065530f (DEFAULT_TABLE table=default.Table_09c35871ae6345f8bd692a8e416323e5 [id=64f9a8f7a56c4a32a12059f106d879c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.089809 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ff0ec53ae71461baf2b8f9ec065530f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.091343 1768 tablet_bootstrap.cc:492] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:17.092731 1444 consensus_peers.cc:597] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 187bc968c2b049028dd25dfb0264c0aa. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:17.092957 1768 tablet_bootstrap.cc:654] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.101891 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 3e65583d0ea84a3ea10ae11462eac663 (DEFAULT_TABLE table=default.Table_c9d965a31a7f4ba3bed33813b02d7dfb [id=f86f7a37b0814f58bcf9e52711fde2a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.102473 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e65583d0ea84a3ea10ae11462eac663. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.103921 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 3e65583d0ea84a3ea10ae11462eac663 (DEFAULT_TABLE table=default.Table_c9d965a31a7f4ba3bed33813b02d7dfb [id=f86f7a37b0814f58bcf9e52711fde2a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.104470 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 3e65583d0ea84a3ea10ae11462eac663 (DEFAULT_TABLE table=default.Table_c9d965a31a7f4ba3bed33813b02d7dfb [id=f86f7a37b0814f58bcf9e52711fde2a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.093456 1775 rpcz_store.cc:276] 1024 19:48:13.610877 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.611209 (+ 332us) service_pool.cc:225] Handling call
1024 19:48:16.926778 (+3315569us) ts_tablet_manager.cc:1151] Deleting tablet cef80e4d14bb4e719bceebf43041d079
1024 19:48:16.926787 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.941931 (+ 15144us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.959320 (+ 17389us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:16.998654 (+ 39334us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3315385}
I20251024 19:48:17.105161 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet cef80e4d14bb4e719bceebf43041d079 (table default.Table_ade7993306ee438baebbf639e2109768 [id=6318d0634b77498a8c1ff8618b8ccf4b]) successfully deleted
I20251024 19:48:17.013298 1776 ts_tablet_manager.cc:1929] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:17.110751 1776 log.cc:1199] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/ad055dc3387b414bb55d09b9683ba17e
I20251024 19:48:17.110998 1768 tablet_bootstrap.cc:492] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:17.111207 1768 ts_tablet_manager.cc:1403] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20251024 19:48:17.111577 1776 ts_tablet_manager.cc:1950] T ad055dc3387b414bb55d09b9683ba17e P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:17.111564 2936 raft_consensus.cc:493] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.111686 2936 raft_consensus.cc:515] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:17.112232 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e65583d0ea84a3ea10ae11462eac663. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.112233 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 797) took 3478 ms. Trace:
I20251024 19:48:17.112385 1775 rpcz_store.cc:276] 1024 19:48:13.633978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.634186 (+ 208us) service_pool.cc:225] Handling call
1024 19:48:17.105532 (+3471346us) ts_tablet_manager.cc:1151] Deleting tablet 73278db2097d4001a538d62dacb97c9b
1024 19:48:17.112085 (+ 6553us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.112211 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3471091}
I20251024 19:48:17.112527 2936 leader_election.cc:290] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:17.113134 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 73278db2097d4001a538d62dacb97c9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73278db2097d4001a538d62dacb97c9b already in progress: creating tablet
I20251024 19:48:17.113674 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 798) took 3473 ms. Trace:
I20251024 19:48:17.113898 1775 rpcz_store.cc:276] 1024 19:48:13.640539 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.640764 (+ 225us) service_pool.cc:225] Handling call
1024 19:48:17.113526 (+3472762us) ts_tablet_manager.cc:1151] Deleting tablet 9377725db312439e8a42dbd46da84a9d
1024 19:48:17.113535 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.113653 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3472541}
W20251024 19:48:17.114076 1308 leader_election.cc:343] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: d9bf08ccf7054fedb9efcbb00354c71a
W20251024 19:48:17.114578 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9377725db312439e8a42dbd46da84a9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9377725db312439e8a42dbd46da84a9d already in progress: creating tablet
I20251024 19:48:17.115198 1766 tablet_bootstrap.cc:492] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
W20251024 19:48:17.116487 1305 leader_election.cc:343] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: d9bf08ccf7054fedb9efcbb00354c71a
I20251024 19:48:17.116616 1305 leader_election.cc:304] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.116865 2936 raft_consensus.cc:2749] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:48:17.116518 1766 ts_tablet_manager.cc:1403] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.177s user 0.006s sys 0.004s
I20251024 19:48:17.117517 1768 raft_consensus.cc:359] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.117715 1768 raft_consensus.cc:385] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.117770 1768 raft_consensus.cc:740] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.118032 1768 consensus_queue.cc:260] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.118561 1768 ts_tablet_manager.cc:1434] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251024 19:48:17.118829 1768 tablet_bootstrap.cc:492] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:17.120309 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 799) took 3450 ms. Trace:
I20251024 19:48:17.120414 1775 rpcz_store.cc:276] 1024 19:48:13.669876 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.670096 (+ 220us) service_pool.cc:225] Handling call
1024 19:48:17.114938 (+3444842us) ts_tablet_manager.cc:1151] Deleting tablet 2764689179504e079924fde7e425dbcd
1024 19:48:17.114948 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.120293 (+ 5345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3444620}
W20251024 19:48:17.121201 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2764689179504e079924fde7e425dbcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2764689179504e079924fde7e425dbcd already in progress: creating tablet
I20251024 19:48:17.121519 1768 tablet_bootstrap.cc:654] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.123709 1775 tablet_replica.cc:333] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:17.124099 1775 raft_consensus.cc:2243] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:17.124758 1775 raft_consensus.cc:2272] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.126127 1775 ts_tablet_manager.cc:1916] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.122696 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76837fecac49493ebd5d24f192b0f693. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.127909 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e65583d0ea84a3ea10ae11462eac663. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.132896 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f33d405c584efaafd992bf9c3f2f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.132683 1766 raft_consensus.cc:359] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.134478 1766 raft_consensus.cc:385] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.135362 1766 raft_consensus.cc:740] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.137612 1183 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet e03c488d0f264684b0897a81a04d7466 (table default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8]). Expected version 2 got 0
I20251024 19:48:17.139117 1185 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_bedd8edc24014953ac8c7dcecf18f128 [id=2ae1c808837e48df97c4efb01a684da9] alter complete (version 1)
I20251024 19:48:17.139294 1766 consensus_queue.cc:260] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.139860 1766 ts_tablet_manager.cc:1434] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251024 19:48:17.142069 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_55339e02660843c3a8536a4b7a06831e" } modify_external_catalogs: true
I20251024 19:48:17.143772 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_81a931b6624742858d05cfc15a78beed" } new_table_name: "default.Table_3c3587f8ff21469a956f564c794672c5" modify_external_catalogs: true
W20251024 19:48:17.148442 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ba845f3f8984585af370c3d39f0ebb9 consensus state has no leader
I20251024 19:48:17.148247 1766 tablet_bootstrap.cc:492] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:17.148717 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a7e76a3cac443ec8d2ab466075aae6b with delete_type TABLET_DATA_DELETED (replaced by tablet 2764689179504e079924fde7e425dbcd) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.148844 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6f21add4230f4d1692ce71616d05ff7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:17.148588 1480 tablet_service.cc:1552] Processing DeleteTablet for tablet 2602335327fc489fa154056c5ef5138f with delete_type TABLET_DATA_DELETED (replaced by tablet 74029a7d7cdc4503a833c1758887b684) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.148926 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 4243f11bf2b34b30b10766c19e2c6409 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:17.149427 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f21add4230f4d1692ce71616d05ff7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.150375 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d129bdc876e425aabcf1c99997e319a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.150871 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet b88a6a5237d146f5b6f082ca3af25b2b with delete_type TABLET_DATA_DELETED (replaced by tablet 639836c656f44a819036f4d00e055bd9) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.151307 1480 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.123524 2936 raft_consensus.cc:493] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.151714 2936 raft_consensus.cc:515] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.150184 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 4193750331874d7ba9bbe6279c3a8e37 with delete_type TABLET_DATA_DELETED (replaced by tablet ec2ffe32f2914ef59d5d77056fbb7d16) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.152338 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet cef80e4d14bb4e719bceebf43041d079 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.152842 2936 leader_election.cc:290] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:17.153146 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca95e470a3db47ceb2da9246e4bc094a consensus state has no leader
I20251024 19:48:17.153223 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 62667092d7e244cca1020ec220ea9124 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:17.153527 1308 leader_election.cc:343] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: f51c56337ebb450aaf62dcc81ef63ae8
I20251024 19:48:17.153499 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3219a7b4a2474607bd3df6f6d3e35642" } modify_external_catalogs: true
I20251024 19:48:17.153671 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3219a7b4a2474607bd3df6f6d3e35642" } modify_external_catalogs: true
I20251024 19:48:17.154173 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f51c56337ebb450aaf62dcc81ef63ae8" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:17.154351 1643 raft_consensus.cc:2468] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:17.154776 2973 raft_consensus.cc:493] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.154906 2973 raft_consensus.cc:515] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.155781 2973 leader_election.cc:290] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.155988 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3324755a5a184a609fae5f7a25d58e4c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:17.156419 1444 leader_election.cc:343] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.156865 1305 leader_election.cc:304] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
I20251024 19:48:17.157285 2936 raft_consensus.cc:2804] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:17.157415 2936 raft_consensus.cc:493] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:17.157545 2936 raft_consensus.cc:3060] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.158190 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3324755a5a184a609fae5f7a25d58e4c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:17.158864 1447 leader_election.cc:343] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.159039 1447 leader_election.cc:304] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.159266 2973 raft_consensus.cc:2749] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.159966 1766 tablet_bootstrap.cc:654] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.159981 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_55339e02660843c3a8536a4b7a06831e" } modify_external_catalogs: true
I20251024 19:48:17.160130 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 607ae453221d422e9d07882830c02275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.160444 2936 raft_consensus.cc:515] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.160584 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet b1fa439afe5a48a79557019687319af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.161022 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f with delete_type TABLET_DATA_DELETED (replaced by tablet 9a1031d785b646b095ece9be666d2a77) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:17.161672 1308 leader_election.cc:343] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: f51c56337ebb450aaf62dcc81ef63ae8
I20251024 19:48:17.162014 2936 leader_election.cc:290] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:17.165014 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b8d30b2fdeec410fa582a05db0968ebb
I20251024 19:48:17.165827 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f51c56337ebb450aaf62dcc81ef63ae8" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:17.165836 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_658377c2e7954211a0e56c012dc08bf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:17.165988 1643 raft_consensus.cc:3060] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:17.166352 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_658377c2e7954211a0e56c012dc08bf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.167698 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7013504a3c6a4813991ac0e9022f53e1" } new_table_name: "default.Table_6a4162ff35fb4f1285ec130d29c15320" modify_external_catalogs: true
I20251024 19:48:17.169075 1643 raft_consensus.cc:2468] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:17.169121 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet b8d30b2fdeec410fa582a05db0968ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.170186 1191 catalog_manager.cc:5649] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:17.170919 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 067631b128fd4a1e9ab72c9507dab6c4 (table default.Table_814a6dad46d54945b70e652b7e0f0032 [id=3f394c59d82b44928beb22f42312a4a3]). Expected version 1 got 0
I20251024 19:48:17.171212 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet d6de2c21796f4bc4904f88b654c3baec (table default.Table_1a079b981b694617b3baf9897dfb37ae [id=18ec057e920e4c21ae8cbe298058b6f3]). Expected version 2 got 0
I20251024 19:48:17.172205 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 804) took 3534 ms. Trace:
I20251024 19:48:17.172457 1305 leader_election.cc:304] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
I20251024 19:48:17.172891 2936 raft_consensus.cc:2804] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:17.174219 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet b8d30b2fdeec410fa582a05db0968ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.175381 1766 tablet_bootstrap.cc:492] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:17.175562 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet b8d30b2fdeec410fa582a05db0968ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.175648 1766 ts_tablet_manager.cc:1403] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.001s
I20251024 19:48:17.178558 1766 raft_consensus.cc:359] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.178884 1766 raft_consensus.cc:385] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.179013 1766 raft_consensus.cc:740] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.179452 1766 consensus_queue.cc:260] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.181473 1766 ts_tablet_manager.cc:1434] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251024 19:48:17.183748 1768 tablet_bootstrap.cc:492] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:17.185299 1768 ts_tablet_manager.cc:1403] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.067s user 0.001s sys 0.003s
I20251024 19:48:17.186247 1768 raft_consensus.cc:359] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:17.187323 2776 raft_consensus.cc:359] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Replica starting. Triggering 77 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:48:17.190429 1768 raft_consensus.cc:385] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.190662 1768 raft_consensus.cc:740] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.191118 1768 consensus_queue.cc:260] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:17.193130 2973 raft_consensus.cc:493] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.193287 2973 raft_consensus.cc:515] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.194015 2973 leader_election.cc:290] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.194406 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edb49884389c4dba957f0ee5af4ae0e4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:17.194615 1643 raft_consensus.cc:2468] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:17.194810 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edb49884389c4dba957f0ee5af4ae0e4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:17.195161 1444 leader_election.cc:304] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:17.195369 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_303e87d506b64ecdbb134e0476487dec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:17.195912 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_303e87d506b64ecdbb134e0476487dec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.196365 2973 raft_consensus.cc:2804] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:17.196448 2973 raft_consensus.cc:493] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:17.196512 2973 raft_consensus.cc:3060] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:17.198104 1447 leader_election.cc:343] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.198271 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet eb6b7df6176e4bccacf7a9059cee5477
I20251024 19:48:17.198694 1372 raft_consensus.cc:3060] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.199259 2973 raft_consensus.cc:515] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.200204 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edb49884389c4dba957f0ee5af4ae0e4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:17.200381 1643 raft_consensus.cc:3060] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.200719 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7b543dc2852946a78c82d145655196ae" } new_table_name: "default.Table_7d02c08e475144cea4a2e808aee8278f" modify_external_catalogs: true
I20251024 19:48:17.203248 1643 raft_consensus.cc:2468] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:17.204097 1444 leader_election.cc:304] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:17.204373 2858 raft_consensus.cc:2804] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:17.204504 2858 raft_consensus.cc:697] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:17.204813 2858 consensus_queue.cc:237] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.205166 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 9489a01434a24d9ca94791252950725c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.172329 1776 rpcz_store.cc:276] 1024 19:48:13.637523 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.763071 (+125548us) service_pool.cc:225] Handling call
1024 19:48:16.950532 (+3187461us) ts_tablet_manager.cc:1151] Deleting tablet ad055dc3387b414bb55d09b9683ba17e
1024 19:48:16.950542 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:16.956144 (+ 5602us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.013198 (+ 57054us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.172183 (+158985us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3187229}
I20251024 19:48:17.206043 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 799) took 3568 ms. Trace:
I20251024 19:48:17.206118 1776 rpcz_store.cc:276] 1024 19:48:13.637142 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.751210 (+114068us) service_pool.cc:225] Handling call
1024 19:48:17.205922 (+3454712us) ts_tablet_manager.cc:1151] Deleting tablet fce7de9523fb4878a58aa8aef0c6b182
1024 19:48:17.205932 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.206026 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3292400}
I20251024 19:48:17.206331 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 639836c656f44a819036f4d00e055bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.206387 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 805) took 3568 ms. Trace:
I20251024 19:48:17.206449 2920 consensus_queue.cc:1048] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:17.210527 2857 raft_consensus.cc:493] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.210641 2857 raft_consensus.cc:515] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.211215 2857 leader_election.cc:290] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.212913 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet eb6b7df6176e4bccacf7a9059cee5477 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.212888 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_088895c10fa249e488c6cd194ea431fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:17.213721 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet ad055dc3387b414bb55d09b9683ba17e (table default.Table_3e341daa717746478259c784488beecf [id=c3de6937e93a48dcb88cabee59a09d32]) successfully deleted
W20251024 19:48:17.214059 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet fce7de9523fb4878a58aa8aef0c6b182 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fce7de9523fb4878a58aa8aef0c6b182 already in progress: creating tablet
I20251024 19:48:17.217926 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50cea450eda9485daba40bf875a1b90c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:17.218502 1444 leader_election.cc:343] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.221897 2973 leader_election.cc:290] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.223681 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50cea450eda9485daba40bf875a1b90c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:17.224277 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edb49884389c4dba957f0ee5af4ae0e4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:17.224817 1447 leader_election.cc:343] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.224988 1447 leader_election.cc:304] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:17.225054 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d610b007a0054067ab8a1278580a98d6 consensus state has no leader
W20251024 19:48:17.225450 1447 leader_election.cc:343] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.225729 2973 raft_consensus.cc:2749] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.225648 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_074163dbc16146cf81fc6db79bd636b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:17.226351 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_074163dbc16146cf81fc6db79bd636b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.227334 1766 tablet_bootstrap.cc:492] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:17.206447 1776 rpcz_store.cc:276] 1024 19:48:13.637624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.763326 (+125702us) service_pool.cc:225] Handling call
1024 19:48:17.206299 (+3442973us) ts_tablet_manager.cc:1151] Deleting tablet b22b58404b8b441ba97ef536885a66e0
1024 19:48:17.206306 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.206374 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3241039}
I20251024 19:48:17.228624 1776 tablet_replica.cc:333] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:17.228977 1776 raft_consensus.cc:2243] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:17.229219 1776 raft_consensus.cc:2272] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.229802 1776 ts_tablet_manager.cc:1916] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.229918 1766 tablet_bootstrap.cc:654] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:17.232307 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b22b58404b8b441ba97ef536885a66e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b22b58404b8b441ba97ef536885a66e0 already in progress: creating tablet
I20251024 19:48:17.236045 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_32da551817ca47239b06d0c761a20839"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:17.237002 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32da551817ca47239b06d0c761a20839 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.237846 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_dbb22386d4e04757bc3b5e73e46c6e1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:17.237942 1305 consensus_peers.cc:597] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce8c179e98944df0b510a0e8daf3920f. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:17.238214 2973 raft_consensus.cc:493] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.238358 2973 raft_consensus.cc:515] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:17.238446 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbb22386d4e04757bc3b5e73e46c6e1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.239598 2973 leader_election.cc:290] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.240017 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2602335327fc489fa154056c5ef5138f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:17.240520 1444 leader_election.cc:343] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.241489 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2602335327fc489fa154056c5ef5138f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:17.242117 1447 leader_election.cc:343] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.242215 1447 leader_election.cc:304] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.242476 2936 raft_consensus.cc:697] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:17.242631 2973 raft_consensus.cc:2749] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.242926 2936 consensus_queue.cc:237] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.251653 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 40338a7939be4240b44dc54cc77ef020
I20251024 19:48:17.253082 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 40338a7939be4240b44dc54cc77ef020 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.255092 1767 tablet_bootstrap.cc:492] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:17.255313 1767 ts_tablet_manager.cc:1403] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.240s user 0.007s sys 0.003s
I20251024 19:48:17.256186 1767 raft_consensus.cc:359] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.258066 1775 ts_tablet_manager.cc:1929] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251024 19:48:17.213531 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_088895c10fa249e488c6cd194ea431fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.261894 1767 raft_consensus.cc:385] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.262203 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet e29abfa3bc8944aea7d1acd03e3eb897
I20251024 19:48:17.262253 1767 raft_consensus.cc:740] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.263041 1767 consensus_queue.cc:260] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.264354 1775 log.cc:1199] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/07e75c7f35544bec82cb994dea5ee4ef
I20251024 19:48:17.264873 1775 ts_tablet_manager.cc:1950] T 07e75c7f35544bec82cb994dea5ee4ef P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:17.263605 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet e29abfa3bc8944aea7d1acd03e3eb897 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.265208 2920 raft_consensus.cc:493] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.265341 2920 raft_consensus.cc:515] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.266263 2920 leader_election.cc:290] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.270694 1768 ts_tablet_manager.cc:1434] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.085s user 0.002s sys 0.000s
I20251024 19:48:17.271121 1768 tablet_bootstrap.cc:492] T c512ec26e3874cd7869216b1fb90fa54 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:17.274192 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3324755a5a184a609fae5f7a25d58e4c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:17.275172 1586 leader_election.cc:343] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.280740 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 187bc968c2b049028dd25dfb0264c0aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.280812 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet c82f60819e7a442bbc0cc04659e14e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.280970 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d129bdc876e425aabcf1c99997e319a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.281098 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 3887fc21726846279fecb02672b9bc07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.281256 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.281436 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 639836c656f44a819036f4d00e055bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.288687 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet e03c488d0f264684b0897a81a04d7466 (table default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8]). Expected version 2 got 0
I20251024 19:48:17.293803 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 4193750331874d7ba9bbe6279c3a8e37 with delete_type TABLET_DATA_DELETED (replaced by tablet ec2ffe32f2914ef59d5d77056fbb7d16) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.294368 1767 ts_tablet_manager.cc:1434] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20251024 19:48:17.294893 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet f83b9290f00145a780318a43249eec12 with delete_type TABLET_DATA_DELETED (replaced by tablet 41176885447e42409ee4363bb4d903f2) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.271679 1766 tablet_bootstrap.cc:492] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:17.295521 1766 ts_tablet_manager.cc:1403] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.003s
I20251024 19:48:17.296255 1767 tablet_bootstrap.cc:492] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:17.296434 1766 raft_consensus.cc:359] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.296615 1766 raft_consensus.cc:385] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.296666 1766 raft_consensus.cc:740] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.296849 1766 consensus_queue.cc:260] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.297394 1766 ts_tablet_manager.cc:1434] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:17.297719 1766 tablet_bootstrap.cc:492] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:17.297988 1767 tablet_bootstrap.cc:654] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.299304 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 805) took 3152 ms. Trace:
I20251024 19:48:17.299376 1775 rpcz_store.cc:276] 1024 19:48:14.147218 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.147278 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:17.120929 (+2973651us) ts_tablet_manager.cc:1151] Deleting tablet 07e75c7f35544bec82cb994dea5ee4ef
1024 19:48:17.123633 (+ 2704us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.239062 (+115429us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.257939 (+ 18877us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.299284 (+ 41345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2973459}
I20251024 19:48:17.299652 1766 tablet_bootstrap.cc:654] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.273204 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3324755a5a184a609fae5f7a25d58e4c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:17.300822 1505 raft_consensus.cc:2468] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:17.301399 1586 leader_election.cc:304] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:17.301739 2926 raft_consensus.cc:2804] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:17.301822 2926 raft_consensus.cc:493] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:17.301901 2926 raft_consensus.cc:3060] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.302448 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 806) took 3150 ms. Trace:
I20251024 19:48:17.302533 1775 rpcz_store.cc:276] 1024 19:48:14.151722 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.151803 (+ 81us) service_pool.cc:225] Handling call
1024 19:48:17.302350 (+3150547us) ts_tablet_manager.cc:1151] Deleting tablet 9fffc42ddda14bd28800917ff2f0ced5
1024 19:48:17.302359 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.302430 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3142825}
I20251024 19:48:17.304843 2926 raft_consensus.cc:515] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.306339 1776 ts_tablet_manager.cc:1929] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251024 19:48:17.306459 1776 log.cc:1199] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/2346c03802a245a2af8c086b626d660d
I20251024 19:48:17.306846 1776 ts_tablet_manager.cc:1950] T 2346c03802a245a2af8c086b626d660d P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:17.306975 2926 leader_election.cc:290] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.307183 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 815) took 3616 ms. Trace:
I20251024 19:48:17.307268 1776 rpcz_store.cc:276] 1024 19:48:13.690621 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.071041 (+380420us) service_pool.cc:225] Handling call
1024 19:48:17.228564 (+3157523us) ts_tablet_manager.cc:1151] Deleting tablet 2346c03802a245a2af8c086b626d660d
1024 19:48:17.228578 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.259002 (+ 30424us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.306216 (+ 47214us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.307164 (+ 948us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3157285}
I20251024 19:48:17.307664 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 816) took 3616 ms. Trace:
I20251024 19:48:17.307763 1776 rpcz_store.cc:276] 1024 19:48:13.690699 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.071292 (+380593us) service_pool.cc:225] Handling call
1024 19:48:17.307542 (+3236250us) ts_tablet_manager.cc:1151] Deleting tablet 73278db2097d4001a538d62dacb97c9b
1024 19:48:17.307551 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.307647 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3236094}
I20251024 19:48:17.307885 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 2346c03802a245a2af8c086b626d660d (table default.Table_1865562320ef412799f93a15c66ade0c [id=aadca5a1b6d4494aa37f8a8cc76ad12a]) successfully deleted
W20251024 19:48:17.308223 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 73278db2097d4001a538d62dacb97c9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73278db2097d4001a538d62dacb97c9b already in progress: creating tablet
I20251024 19:48:17.308231 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3324755a5a184a609fae5f7a25d58e4c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:17.308699 1586 leader_election.cc:343] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.309235 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 807) took 3035 ms. Trace:
I20251024 19:48:17.309362 1775 rpcz_store.cc:276] 1024 19:48:14.273485 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.274080 (+ 595us) service_pool.cc:225] Handling call
1024 19:48:17.309028 (+3034948us) ts_tablet_manager.cc:1151] Deleting tablet b74415cf7c7e47a696b454a5d44d8017
1024 19:48:17.309040 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.309213 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3034733}
I20251024 19:48:17.312970 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3324755a5a184a609fae5f7a25d58e4c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:17.317312 1507 raft_consensus.cc:3060] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.317476 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 808) took 3043 ms. Trace:
I20251024 19:48:17.317565 1775 rpcz_store.cc:276] 1024 19:48:14.273654 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.275473 (+ 1819us) service_pool.cc:225] Handling call
1024 19:48:17.317358 (+3041885us) ts_tablet_manager.cc:1151] Deleting tablet b19cce078dc7426aacbb60f45379e506
1024 19:48:17.317367 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.317461 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3041690}
I20251024 19:48:17.318071 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 809) took 3030 ms. Trace:
I20251024 19:48:17.318161 1775 rpcz_store.cc:276] 1024 19:48:14.288046 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.288238 (+ 192us) service_pool.cc:225] Handling call
1024 19:48:17.317981 (+3029743us) ts_tablet_manager.cc:1151] Deleting tablet d76410afd7ba4c73836f4791ef739056
1024 19:48:17.317993 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.318057 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3029537}
I20251024 19:48:17.318591 3043 tablet.cc:1722] T 187925082cdd4923a6e9148bf9f270fd P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:17.319067 1775 tablet_replica.cc:333] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:17.319257 1775 raft_consensus.cc:2243] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:17.319417 1775 raft_consensus.cc:2272] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.320348 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 817) took 3629 ms. Trace:
I20251024 19:48:17.320550 1766 tablet_bootstrap.cc:492] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:17.320446 1776 rpcz_store.cc:276] 1024 19:48:13.690759 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.071460 (+380701us) service_pool.cc:225] Handling call
1024 19:48:17.320217 (+3248757us) ts_tablet_manager.cc:1151] Deleting tablet 9377725db312439e8a42dbd46da84a9d
1024 19:48:17.320226 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.320325 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3248549}
I20251024 19:48:17.320749 1766 ts_tablet_manager.cc:1403] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20251024 19:48:17.320927 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 818) took 3630 ms. Trace:
I20251024 19:48:17.321009 1776 rpcz_store.cc:276] 1024 19:48:13.690838 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.071676 (+380838us) service_pool.cc:225] Handling call
1024 19:48:17.320822 (+3249146us) ts_tablet_manager.cc:1151] Deleting tablet 2764689179504e079924fde7e425dbcd
1024 19:48:17.320832 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.320911 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3249006}
I20251024 19:48:17.321755 1766 raft_consensus.cc:359] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.321995 1766 raft_consensus.cc:385] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.322140 1776 tablet_replica.cc:333] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:17.322248 1766 raft_consensus.cc:740] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.322338 1776 raft_consensus.cc:2243] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:17.322713 1776 raft_consensus.cc:2272] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.322692 1766 consensus_queue.cc:260] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:17.323048 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9377725db312439e8a42dbd46da84a9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9377725db312439e8a42dbd46da84a9d already in progress: creating tablet
W20251024 19:48:17.323428 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2764689179504e079924fde7e425dbcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2764689179504e079924fde7e425dbcd already in progress: creating tablet
I20251024 19:48:17.323740 1766 ts_tablet_manager.cc:1434] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:17.324210 1766 tablet_bootstrap.cc:492] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:17.324357 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b724eadbb86e4eeab4ba8dfd565290ff consensus state has no leader
I20251024 19:48:17.325408 1776 ts_tablet_manager.cc:1916] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.326972 1766 tablet_bootstrap.cc:654] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.328269 1507 raft_consensus.cc:2468] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:17.333040 3042 tablet.cc:1722] T 187925082cdd4923a6e9148bf9f270fd P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:17.336911 1586 leader_election.cc:304] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:17.337512 2920 raft_consensus.cc:2804] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:17.337711 2920 raft_consensus.cc:697] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:17.337796 2858 raft_consensus.cc:493] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.338024 2858 raft_consensus.cc:515] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.339192 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d01572e51694d9bb58c3a59d4205102" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:17.339741 1444 leader_election.cc:343] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:17.341399 1447 consensus_peers.cc:597] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:17.341743 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d01572e51694d9bb58c3a59d4205102" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:17.342284 1447 leader_election.cc:343] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.342381 1447 leader_election.cc:304] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.342602 2973 raft_consensus.cc:2749] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.342644 2858 leader_election.cc:290] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.343266 1768 tablet_bootstrap.cc:654] T c512ec26e3874cd7869216b1fb90fa54 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.343919 1767 tablet_bootstrap.cc:492] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:17.344098 1767 ts_tablet_manager.cc:1403] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.003s
I20251024 19:48:17.322007 1775 ts_tablet_manager.cc:1916] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.345073 1767 raft_consensus.cc:359] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.345285 1767 raft_consensus.cc:385] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.345377 1767 raft_consensus.cc:740] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.345664 1767 consensus_queue.cc:260] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.346203 1767 ts_tablet_manager.cc:1434] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:17.346606 1767 tablet_bootstrap.cc:492] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:17.338032 2920 consensus_queue.cc:237] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.351801 1767 tablet_bootstrap.cc:654] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.353350 1776 ts_tablet_manager.cc:1929] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:17.353549 1776 log.cc:1199] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/d38154d3ffd34f4b8d7305b1c23a4fae
I20251024 19:48:17.354269 1776 ts_tablet_manager.cc:1950] T d38154d3ffd34f4b8d7305b1c23a4fae P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:17.355953 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 819) took 3599 ms. Trace:
I20251024 19:48:17.356200 1776 rpcz_store.cc:276] 1024 19:48:13.756943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.071825 (+314882us) service_pool.cc:225] Handling call
1024 19:48:17.322093 (+3250268us) ts_tablet_manager.cc:1151] Deleting tablet d38154d3ffd34f4b8d7305b1c23a4fae
1024 19:48:17.322104 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.334891 (+ 12787us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.353252 (+ 18361us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.355902 (+ 2650us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3250093}
I20251024 19:48:17.357085 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet d38154d3ffd34f4b8d7305b1c23a4fae (table default.Table_a395ff27a51a4b5e878e524825bdd2cf [id=ed879934f5754b92b8cff234f3f66333]) successfully deleted
I20251024 19:48:17.357445 1776 tablet_replica.cc:333] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:17.358408 1776 raft_consensus.cc:2243] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:17.359196 1776 raft_consensus.cc:2272] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.361052 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet f51c56337ebb450aaf62dcc81ef63ae8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.361711 1776 ts_tablet_manager.cc:1916] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.364418 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 505e4fb0755a4caaa462e5f5154bfee2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b39e014607b4437b255c26c92c71861) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.369863 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet eb6b7df6176e4bccacf7a9059cee5477 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.370738 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet eb6b7df6176e4bccacf7a9059cee5477 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.373376 2920 consensus_queue.cc:1048] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:17.378423 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e7255804242d424f8cbf89ef57d4ff9f consensus state has no leader
I20251024 19:48:17.380919 2858 raft_consensus.cc:493] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.381048 2858 raft_consensus.cc:515] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.382026 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f630fd412ca4c4f8570359c1678e6af" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:17.382490 1444 leader_election.cc:343] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.382637 1766 tablet_bootstrap.cc:492] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:17.382812 1766 ts_tablet_manager.cc:1403] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.003s
I20251024 19:48:17.382889 2858 leader_election.cc:290] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.383481 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f630fd412ca4c4f8570359c1678e6af" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:17.383741 1766 raft_consensus.cc:359] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.383935 1766 raft_consensus.cc:385] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251024 19:48:17.383985 1447 leader_election.cc:343] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.384089 1447 leader_election.cc:304] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.384004 1766 raft_consensus.cc:740] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.384518 2858 raft_consensus.cc:2749] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.384537 1766 consensus_queue.cc:260] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.386377 1766 ts_tablet_manager.cc:1434] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251024 19:48:17.388425 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 3324755a5a184a609fae5f7a25d58e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.391458 1767 tablet_bootstrap.cc:492] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:17.391809 1767 ts_tablet_manager.cc:1403] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.008s
I20251024 19:48:17.394627 1775 ts_tablet_manager.cc:1929] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:17.394770 1775 log.cc:1199] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/d38154d3ffd34f4b8d7305b1c23a4fae
I20251024 19:48:17.395171 1775 ts_tablet_manager.cc:1950] T d38154d3ffd34f4b8d7305b1c23a4fae P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:17.396194 1767 raft_consensus.cc:359] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:17.396593 1767 raft_consensus.cc:385] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.396792 1767 raft_consensus.cc:740] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.397087 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet b240965605904533bd2af99f61aaef26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.397321 1767 consensus_queue.cc:260] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:17.397882 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f630fd412ca4c4f8570359c1678e6af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.398525 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet cb0405765c6146cbbb187af96a60be2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.399123 1776 ts_tablet_manager.cc:1929] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:17.399370 2858 consensus_queue.cc:1048] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:17.399623 1776 log.cc:1199] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/505e4fb0755a4caaa462e5f5154bfee2
I20251024 19:48:17.399844 1767 ts_tablet_manager.cc:1434] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251024 19:48:17.401053 1767 tablet_bootstrap.cc:492] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:17.401335 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 844638a6b6944d0394df3e78a8ea51d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.401490 1776 ts_tablet_manager.cc:1950] T 505e4fb0755a4caaa462e5f5154bfee2 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:17.401794 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f89a3bbeb9b4d57a49078729110d951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.403074 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 40338a7939be4240b44dc54cc77ef020 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.403146 1768 tablet_bootstrap.cc:492] T c512ec26e3874cd7869216b1fb90fa54 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:17.403250 1768 ts_tablet_manager.cc:1403] T c512ec26e3874cd7869216b1fb90fa54 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.132s user 0.006s sys 0.004s
I20251024 19:48:17.403541 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 40338a7939be4240b44dc54cc77ef020 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.404085 1768 raft_consensus.cc:359] T c512ec26e3874cd7869216b1fb90fa54 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.405675 1768 raft_consensus.cc:385] T c512ec26e3874cd7869216b1fb90fa54 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.406744 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 820) took 3649 ms. Trace:
I20251024 19:48:17.408128 1768 raft_consensus.cc:740] T c512ec26e3874cd7869216b1fb90fa54 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.407260 1776 rpcz_store.cc:276] 1024 19:48:13.757140 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.072023 (+314883us) service_pool.cc:225] Handling call
1024 19:48:17.357180 (+3285157us) ts_tablet_manager.cc:1151] Deleting tablet 505e4fb0755a4caaa462e5f5154bfee2
1024 19:48:17.357244 (+ 64us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.377564 (+ 20320us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.398541 (+ 20977us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.406688 (+ 8147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3284902}
I20251024 19:48:17.409025 1557 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:17.409111 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 505e4fb0755a4caaa462e5f5154bfee2 (table default.Table_74e4f24f12db403686548924c83c5191 [id=508d6cdc5d4a494bad31eaf36383e214]) successfully deleted
I20251024 19:48:17.410411 1766 tablet_bootstrap.cc:492] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:17.408424 1768 consensus_queue.cc:260] T c512ec26e3874cd7869216b1fb90fa54 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.411798 1768 ts_tablet_manager.cc:1434] T c512ec26e3874cd7869216b1fb90fa54 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251024 19:48:17.412254 1766 tablet_bootstrap.cc:654] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.412163 3051 tablet.cc:1722] T 187925082cdd4923a6e9148bf9f270fd P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:17.414280 3029 raft_consensus.cc:493] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.414482 3029 raft_consensus.cc:515] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:17.415149 3029 leader_election.cc:290] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:17.416400 1768 tablet_bootstrap.cc:492] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:17.417097 1305 leader_election.cc:343] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: d38154d3ffd34f4b8d7305b1c23a4fae
I20251024 19:48:17.419559 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 810) took 3131 ms. Trace:
I20251024 19:48:17.420722 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet e29abfa3bc8944aea7d1acd03e3eb897 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.421761 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet e29abfa3bc8944aea7d1acd03e3eb897 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:17.422909 1308 leader_election.cc:343] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: d38154d3ffd34f4b8d7305b1c23a4fae
I20251024 19:48:17.423029 1308 leader_election.cc:304] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.423424 2936 raft_consensus.cc:2749] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:17.410650 1586 consensus_peers.cc:597] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:17.410163 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 806) took 3720 ms. Trace:
I20251024 19:48:17.424367 1776 rpcz_store.cc:276] 1024 19:48:13.689925 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.861261 (+171336us) service_pool.cc:225] Handling call
1024 19:48:17.409421 (+3548160us) ts_tablet_manager.cc:1151] Deleting tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4
1024 19:48:17.409607 (+ 186us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.410143 (+ 536us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3319228}
W20251024 19:48:17.426088 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4 already in progress: creating tablet
I20251024 19:48:17.420002 1775 rpcz_store.cc:276] 1024 19:48:14.288183 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.288483 (+ 300us) service_pool.cc:225] Handling call
1024 19:48:17.319021 (+3030538us) ts_tablet_manager.cc:1151] Deleting tablet d38154d3ffd34f4b8d7305b1c23a4fae
1024 19:48:17.319030 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.373004 (+ 53974us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.394511 (+ 21507us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.419430 (+ 24919us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3030375}
I20251024 19:48:17.426640 1776 tablet_replica.cc:333] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:17.427161 1767 tablet_bootstrap.cc:654] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.427896 3043 tablet.cc:1722] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:17.428537 1776 raft_consensus.cc:2243] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:17.429332 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 07e75c7f35544bec82cb994dea5ee4ef (table default.Table_83b4409228694c939a4a734c3b6bdd76 [id=450dc07c78d343aea5c8c2d2a6e26444]) successfully deleted
I20251024 19:48:17.429548 1776 raft_consensus.cc:2272] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:17.429836 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9fffc42ddda14bd28800917ff2f0ced5 because the tablet was not found. No further retry: Not found: Tablet not found: 9fffc42ddda14bd28800917ff2f0ced5
W20251024 19:48:17.430327 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b74415cf7c7e47a696b454a5d44d8017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b74415cf7c7e47a696b454a5d44d8017 already in progress: creating tablet
W20251024 19:48:17.431012 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b19cce078dc7426aacbb60f45379e506 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b19cce078dc7426aacbb60f45379e506 already in progress: creating tablet
W20251024 19:48:17.431501 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet d76410afd7ba4c73836f4791ef739056 because the tablet was not found. No further retry: Not found: Tablet not found: d76410afd7ba4c73836f4791ef739056
I20251024 19:48:17.432035 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet d38154d3ffd34f4b8d7305b1c23a4fae with delete_type TABLET_DATA_DELETED (replaced by tablet 68ca2dd438d94bf0b5f1fc69f0a94517) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.432049 1776 ts_tablet_manager.cc:1916] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.432199 3042 tablet.cc:1722] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:17.439402 1768 tablet_bootstrap.cc:654] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.439718 2857 raft_consensus.cc:493] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.439857 2857 raft_consensus.cc:515] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:17.440506 2857 leader_election.cc:290] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:17.441381 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b240965605904533bd2af99f61aaef26" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:17.441661 1643 raft_consensus.cc:2468] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:17.442111 1444 leader_election.cc:304] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:17.442453 2857 raft_consensus.cc:2804] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:17.442531 2857 raft_consensus.cc:493] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:17.442597 2857 raft_consensus.cc:3060] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.442901 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 811) took 3153 ms. Trace:
I20251024 19:48:17.445716 2857 raft_consensus.cc:515] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:17.443900 1775 rpcz_store.cc:276] 1024 19:48:14.289078 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.289355 (+ 277us) service_pool.cc:225] Handling call
1024 19:48:17.441525 (+3152170us) ts_tablet_manager.cc:1151] Deleting tablet b6661b9e9a174849b6093d09cd8c1251
1024 19:48:17.441540 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.442577 (+ 1037us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3151674}
I20251024 19:48:17.446926 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b240965605904533bd2af99f61aaef26" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:17.447140 2857 leader_election.cc:290] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:17.447088 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b240965605904533bd2af99f61aaef26" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:17.447273 1642 raft_consensus.cc:3060] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.447824 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b240965605904533bd2af99f61aaef26" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:17.449896 1447 leader_election.cc:343] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:17.450104 1447 leader_election.cc:343] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.450223 1642 raft_consensus.cc:2468] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:17.450690 1444 leader_election.cc:304] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:17.451126 2857 raft_consensus.cc:2804] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:17.451387 2857 raft_consensus.cc:697] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:17.451699 2857 consensus_queue.cc:237] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:17.452607 1775 tablet_replica.cc:333] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:17.459503 1775 raft_consensus.cc:2243] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:17.465134 1775 raft_consensus.cc:2272] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.473274 1775 ts_tablet_manager.cc:1916] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.474926 1768 tablet_bootstrap.cc:492] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:17.475142 1768 ts_tablet_manager.cc:1403] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.002s
I20251024 19:48:17.476236 1768 raft_consensus.cc:359] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.476509 1768 raft_consensus.cc:385] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.476622 1768 raft_consensus.cc:740] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.476933 1768 consensus_queue.cc:260] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.477697 1768 ts_tablet_manager.cc:1434] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:17.478021 1768 tablet_bootstrap.cc:492] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:17.478804 2973 raft_consensus.cc:493] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.478945 2973 raft_consensus.cc:515] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.479560 2973 leader_election.cc:290] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.480269 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067631b128fd4a1e9ab72c9507dab6c4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:17.480857 1444 leader_election.cc:343] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.481966 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067631b128fd4a1e9ab72c9507dab6c4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:17.482540 1447 leader_election.cc:343] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.482723 1447 leader_election.cc:304] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.482939 2973 raft_consensus.cc:2749] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:17.491370 1308 consensus_peers.cc:597] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2346c03802a245a2af8c086b626d660d. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:17.492628 1766 tablet_bootstrap.cc:492] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:17.492774 1766 ts_tablet_manager.cc:1403] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.082s user 0.005s sys 0.000s
I20251024 19:48:17.493629 1766 raft_consensus.cc:359] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.493824 1766 raft_consensus.cc:385] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.493875 1766 raft_consensus.cc:740] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.494069 1766 consensus_queue.cc:260] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.494570 1766 ts_tablet_manager.cc:1434] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:17.494848 1766 tablet_bootstrap.cc:492] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:17.497373 1766 tablet_bootstrap.cc:654] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.499588 1767 tablet_bootstrap.cc:492] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:17.499744 1767 ts_tablet_manager.cc:1403] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.099s user 0.004s sys 0.003s
I20251024 19:48:17.500689 1767 raft_consensus.cc:359] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.500860 1767 raft_consensus.cc:385] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.501116 1767 raft_consensus.cc:740] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.501420 1767 consensus_queue.cc:260] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.504149 1767 ts_tablet_manager.cc:1434] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251024 19:48:17.505155 1767 tablet_bootstrap.cc:492] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:17.507205 1767 tablet_bootstrap.cc:654] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.507668 1186 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet d6de2c21796f4bc4904f88b654c3baec (table default.Table_1a079b981b694617b3baf9897dfb37ae [id=18ec057e920e4c21ae8cbe298058b6f3]). Expected version 2 got 0
I20251024 19:48:17.507824 1776 ts_tablet_manager.cc:1929] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:17.509123 1776 log.cc:1199] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/07e75c7f35544bec82cb994dea5ee4ef
I20251024 19:48:17.509677 1776 ts_tablet_manager.cc:1950] T 07e75c7f35544bec82cb994dea5ee4ef P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:17.510545 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 826) took 3365 ms. Trace:
I20251024 19:48:17.510655 1370 raft_consensus.cc:3060] T c512ec26e3874cd7869216b1fb90fa54 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.510711 1776 rpcz_store.cc:276] 1024 19:48:14.145030 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.145129 (+ 99us) service_pool.cc:225] Handling call
1024 19:48:17.426401 (+3281272us) ts_tablet_manager.cc:1151] Deleting tablet 07e75c7f35544bec82cb994dea5ee4ef
1024 19:48:17.426416 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.463373 (+ 36957us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.507701 (+ 44328us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.510526 (+ 2825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3280892}
I20251024 19:48:17.511286 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 9489a01434a24d9ca94791252950725c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.511360 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 07e75c7f35544bec82cb994dea5ee4ef (table default.Table_83b4409228694c939a4a734c3b6bdd76 [id=450dc07c78d343aea5c8c2d2a6e26444]) successfully deleted
I20251024 19:48:17.511404 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 3324755a5a184a609fae5f7a25d58e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.511572 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 830) took 3200 ms. Trace:
I20251024 19:48:17.511678 1776 rpcz_store.cc:276] 1024 19:48:14.310939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.311042 (+ 103us) service_pool.cc:225] Handling call
1024 19:48:17.511131 (+3200089us) ts_tablet_manager.cc:1151] Deleting tablet 839cef50795840fab564b94d88565f8c
1024 19:48:17.511140 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.511555 (+ 415us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3199855}
I20251024 19:48:17.512167 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 829) took 3201 ms. Trace:
I20251024 19:48:17.512283 1776 rpcz_store.cc:276] 1024 19:48:14.310845 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.311001 (+ 156us) service_pool.cc:225] Handling call
1024 19:48:17.512064 (+3201063us) ts_tablet_manager.cc:1151] Deleting tablet cf18f375266b4448b3f67afd787df6ad
1024 19:48:17.512072 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.512152 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3200627}
I20251024 19:48:17.512758 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 828) took 3202 ms. Trace:
I20251024 19:48:17.512882 1776 rpcz_store.cc:276] 1024 19:48:14.310747 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.311286 (+ 539us) service_pool.cc:225] Handling call
1024 19:48:17.512661 (+3201375us) ts_tablet_manager.cc:1151] Deleting tablet b19cce078dc7426aacbb60f45379e506
1024 19:48:17.512671 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.512744 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3201073}
I20251024 19:48:17.514269 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 827) took 3203 ms. Trace:
I20251024 19:48:17.514411 1776 rpcz_store.cc:276] 1024 19:48:14.310609 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.311252 (+ 643us) service_pool.cc:225] Handling call
1024 19:48:17.513237 (+3201985us) ts_tablet_manager.cc:1151] Deleting tablet b74415cf7c7e47a696b454a5d44d8017
1024 19:48:17.514163 (+ 926us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.514250 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3112937}
I20251024 19:48:17.514919 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 841) took 2858 ms. Trace:
I20251024 19:48:17.515041 1776 rpcz_store.cc:276] 1024 19:48:14.656325 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.656386 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:17.514818 (+2858432us) ts_tablet_manager.cc:1151] Deleting tablet b3e9960b7821421aa23e7ef5389f894a
1024 19:48:17.514829 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.514904 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2798879}
W20251024 19:48:17.515125 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 839cef50795840fab564b94d88565f8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 839cef50795840fab564b94d88565f8c already in progress: creating tablet
W20251024 19:48:17.515374 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet cf18f375266b4448b3f67afd787df6ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf18f375266b4448b3f67afd787df6ad already in progress: creating tablet
I20251024 19:48:17.515616 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 847) took 2793 ms. Trace:
W20251024 19:48:17.515636 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b19cce078dc7426aacbb60f45379e506 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b19cce078dc7426aacbb60f45379e506 already in progress: creating tablet
I20251024 19:48:17.515744 1776 rpcz_store.cc:276] 1024 19:48:14.722178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.722238 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:17.515515 (+2793277us) ts_tablet_manager.cc:1151] Deleting tablet 292c31e7d4ce411bb336a1405ca43ba5
1024 19:48:17.515525 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.515600 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2792973}
W20251024 19:48:17.515825 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b74415cf7c7e47a696b454a5d44d8017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b74415cf7c7e47a696b454a5d44d8017 already in progress: creating tablet
W20251024 19:48:17.516049 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b3e9960b7821421aa23e7ef5389f894a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3e9960b7821421aa23e7ef5389f894a already in progress: creating tablet
W20251024 19:48:17.516389 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 292c31e7d4ce411bb336a1405ca43ba5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 292c31e7d4ce411bb336a1405ca43ba5 already in progress: creating tablet
W20251024 19:48:17.518025 1308 consensus_peers.cc:597] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce8c179e98944df0b510a0e8daf3920f. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:17.516456 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 846) took 2794 ms. Trace:
I20251024 19:48:17.518527 1776 rpcz_store.cc:276] 1024 19:48:14.722025 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.722788 (+ 763us) service_pool.cc:225] Handling call
1024 19:48:17.516334 (+2793546us) ts_tablet_manager.cc:1151] Deleting tablet 7cab9c17496947efbbe3cd9e5d8a49a8
1024 19:48:17.516350 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.516438 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2793247}
I20251024 19:48:17.520082 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 843) took 2832 ms. Trace:
I20251024 19:48:17.520206 1776 rpcz_store.cc:276] 1024 19:48:14.687674 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.688324 (+ 650us) service_pool.cc:225] Handling call
1024 19:48:17.519980 (+2831656us) ts_tablet_manager.cc:1151] Deleting tablet e51e597b8fd044e393a3af07da25586c
1024 19:48:17.519989 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.520066 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2723042}
I20251024 19:48:17.520707 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 849) took 2734 ms. Trace:
I20251024 19:48:17.520875 1776 rpcz_store.cc:276] 1024 19:48:14.786509 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.786568 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:17.520608 (+2734040us) ts_tablet_manager.cc:1151] Deleting tablet 520060a9adca4bb0b8616bcc006dce52
1024 19:48:17.520617 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.520692 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19307,"tablet-delete.queue_time_us":2701988}
I20251024 19:48:17.521967 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 844) took 2834 ms. Trace:
I20251024 19:48:17.522341 1776 rpcz_store.cc:276] 1024 19:48:14.687848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.687908 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:17.521860 (+2833952us) ts_tablet_manager.cc:1151] Deleting tablet 9684072368f14cc2942b4e240f90b0f2
1024 19:48:17.521871 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.521950 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19141,"tablet-delete.queue_time_us":2702978}
I20251024 19:48:17.523087 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 853) took 2353 ms. Trace:
I20251024 19:48:17.523658 1776 rpcz_store.cc:276] 1024 19:48:15.169793 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.170230 (+ 437us) service_pool.cc:225] Handling call
1024 19:48:17.522983 (+2352753us) ts_tablet_manager.cc:1151] Deleting tablet 9f791083b4964ff9b0f5fc575478892c
1024 19:48:17.522992 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.523070 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352570}
I20251024 19:48:17.524369 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 852) took 2354 ms. Trace:
W20251024 19:48:17.524894 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7cab9c17496947efbbe3cd9e5d8a49a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7cab9c17496947efbbe3cd9e5d8a49a8 already in progress: creating tablet
I20251024 19:48:17.525209 1776 rpcz_store.cc:276] 1024 19:48:15.169657 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.181549 (+ 11892us) service_pool.cc:225] Handling call
1024 19:48:17.524273 (+2342724us) ts_tablet_manager.cc:1151] Deleting tablet 95f62010a24f432c84c89345f9b02b2f
1024 19:48:17.524281 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.524353 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2342363}
W20251024 19:48:17.525344 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet e51e597b8fd044e393a3af07da25586c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e51e597b8fd044e393a3af07da25586c already in progress: creating tablet
I20251024 19:48:17.525123 1767 tablet_bootstrap.cc:492] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:17.525653 1767 ts_tablet_manager.cc:1403] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
W20251024 19:48:17.525672 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 520060a9adca4bb0b8616bcc006dce52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 520060a9adca4bb0b8616bcc006dce52 already in progress: creating tablet
W20251024 19:48:17.525964 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9684072368f14cc2942b4e240f90b0f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9684072368f14cc2942b4e240f90b0f2 already in progress: creating tablet
W20251024 19:48:17.526273 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9f791083b4964ff9b0f5fc575478892c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f791083b4964ff9b0f5fc575478892c already in progress: creating tablet
I20251024 19:48:17.526611 1767 raft_consensus.cc:359] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.526844 1767 raft_consensus.cc:385] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.526938 1767 raft_consensus.cc:740] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.527189 1767 consensus_queue.cc:260] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.529765 1767 ts_tablet_manager.cc:1434] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251024 19:48:17.530086 1767 tablet_bootstrap.cc:492] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:17.532107 1767 tablet_bootstrap.cc:654] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.532437 1776 tablet_replica.cc:333] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:17.533528 1776 raft_consensus.cc:2243] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
W20251024 19:48:17.533849 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 95f62010a24f432c84c89345f9b02b2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95f62010a24f432c84c89345f9b02b2f already in progress: creating tablet
I20251024 19:48:17.534020 1776 raft_consensus.cc:2272] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.533463 2776 raft_consensus.cc:740] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8906492b1a4368936fac95c11e1f73, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.534650 1776 ts_tablet_manager.cc:1916] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.541815 1190 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_55f8c66bd97343438051a98af12d8c2c [id=33f5ac73298543d0b5af77a27e897b79] alter complete (version 1)
I20251024 19:48:17.543272 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 16bc737104ab46b3b071a0760dbc847d with delete_type TABLET_DATA_DELETED (replaced by tablet 385e71e672154e399f14321eac0d8625) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.543619 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet b240965605904533bd2af99f61aaef26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.540755 2776 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 517, Last appended: 1.594, Last appended by leader: 594, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:48:17.543823 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet be8126459ef541f39bc133f27c7d0f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.546725 3061 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:48:17.547047 3061 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 19:48:17.548653 1767 tablet_bootstrap.cc:492] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:17.548780 1767 ts_tablet_manager.cc:1403] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20251024 19:48:17.549559 1767 raft_consensus.cc:359] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.549827 1767 raft_consensus.cc:385] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.550050 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 2602335327fc489fa154056c5ef5138f with delete_type TABLET_DATA_DELETED (replaced by tablet 74029a7d7cdc4503a833c1758887b684) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.549937 1767 raft_consensus.cc:740] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.550510 1767 consensus_queue.cc:260] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.552510 1767 ts_tablet_manager.cc:1434] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:17.553681 1767 tablet_bootstrap.cc:492] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:17.555752 1767 tablet_bootstrap.cc:654] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.556962 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 62667092d7e244cca1020ec220ea9124 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.545522 1776 ts_tablet_manager.cc:1929] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251024 19:48:17.557240 1776 log.cc:1199] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/8ba85164015448d4a688a473cce90c5c
I20251024 19:48:17.557755 1776 ts_tablet_manager.cc:1950] T 8ba85164015448d4a688a473cce90c5c P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:17.558552 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 854) took 2284 ms. Trace:
I20251024 19:48:17.558681 1776 rpcz_store.cc:276] 1024 19:48:15.274001 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.276736 (+ 2735us) service_pool.cc:225] Handling call
1024 19:48:17.532395 (+2255659us) ts_tablet_manager.cc:1151] Deleting tablet 8ba85164015448d4a688a473cce90c5c
1024 19:48:17.532404 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.538600 (+ 6196us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.545411 (+ 6811us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.558533 (+ 13122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2252237}
I20251024 19:48:17.559193 1776 tablet_replica.cc:333] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:17.559415 1776 raft_consensus.cc:2243] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:17.559614 1776 raft_consensus.cc:2272] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.560117 1776 ts_tablet_manager.cc:1916] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.560070 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 8ba85164015448d4a688a473cce90c5c (table default.Table_fdf3a5f81257407d8b5d28c7f7cab1b7 [id=0fa321e2d4d847e69435f6de429d8779]) successfully deleted
I20251024 19:48:17.565418 2776 sys_catalog.cc:565] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 19:48:17.565944 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet d38154d3ffd34f4b8d7305b1c23a4fae (table default.Table_a395ff27a51a4b5e878e524825bdd2cf [id=ed879934f5754b92b8cff234f3f66333]) successfully deleted
W20251024 19:48:17.566584 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b6661b9e9a174849b6093d09cd8c1251 because the tablet was not found. No further retry: Not found: Tablet not found: b6661b9e9a174849b6093d09cd8c1251
I20251024 19:48:17.567363 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_023e47eac6c24e328d5a01fcf3b2f579" } modify_external_catalogs: true
I20251024 19:48:17.567574 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_023e47eac6c24e328d5a01fcf3b2f579" } modify_external_catalogs: true
I20251024 19:48:17.568500 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2acb8ac85b4e4eae9ec2e97990416a3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:17.569218 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2acb8ac85b4e4eae9ec2e97990416a3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.575945 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4be8c37e3871442480b6f9c25bb75797" } new_table_name: "default.Table_71777a1b569c44559975f3db7d8cf059" modify_external_catalogs: true
I20251024 19:48:17.577513 1776 ts_tablet_manager.cc:1929] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:17.578204 1776 log.cc:1199] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/7c77f6bbf7b145389ca6cf74ddd3aa3f
I20251024 19:48:17.578563 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 94fc8c5716af47faa4e831f05fce8610
I20251024 19:48:17.578982 1776 ts_tablet_manager.cc:1950] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:17.578970 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d488cc5bda8c4725a45d2a2cadb7fab7" } new_table_name: "default.Table_532ecd1242ea40d29a9d1bcf6fa8a488" modify_external_catalogs: true
I20251024 19:48:17.579499 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 94fc8c5716af47faa4e831f05fce8610 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.580829 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 871) took 2039 ms. Trace:
I20251024 19:48:17.581812 1776 rpcz_store.cc:276] 1024 19:48:15.540922 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.541017 (+ 95us) service_pool.cc:225] Handling call
1024 19:48:17.559152 (+2018135us) ts_tablet_manager.cc:1151] Deleting tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f
1024 19:48:17.559162 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.563552 (+ 4390us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.577311 (+ 13759us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.580807 (+ 3496us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2017898}
I20251024 19:48:17.582842 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f (table default.Table_a1d09325af844d71b290d95b8a4aaf12 [id=b62ec7bcad7c41b1b946e9ebf2f4bf39]) successfully deleted
I20251024 19:48:17.582624 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 870) took 2044 ms. Trace:
I20251024 19:48:17.583153 1776 rpcz_store.cc:276] 1024 19:48:15.537832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.537950 (+ 118us) service_pool.cc:225] Handling call
1024 19:48:17.582416 (+2044466us) ts_tablet_manager.cc:1151] Deleting tablet 584cbe0c396841f794e665aae3ebbd91
1024 19:48:17.582486 (+ 70us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.582605 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2039394}
I20251024 19:48:17.583513 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 872) took 2037 ms. Trace:
I20251024 19:48:17.583572 1696 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:17.583600 1776 rpcz_store.cc:276] 1024 19:48:15.546522 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.546611 (+ 89us) service_pool.cc:225] Handling call
1024 19:48:17.583410 (+2036799us) ts_tablet_manager.cc:1151] Deleting tablet 452f77091adc4cf09de2231f2f6d845b
1024 19:48:17.583420 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.583497 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2035235}
W20251024 19:48:17.583946 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 584cbe0c396841f794e665aae3ebbd91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 584cbe0c396841f794e665aae3ebbd91 already in progress: creating tablet
I20251024 19:48:17.584689 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 873) took 2026 ms. Trace:
I20251024 19:48:17.586669 1776 rpcz_store.cc:276] 1024 19:48:15.558220 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.558266 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:17.584574 (+2026308us) ts_tablet_manager.cc:1151] Deleting tablet 2cbeaac19e834ecf98e5b8afc952b9da
1024 19:48:17.584583 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.584663 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2026114}
I20251024 19:48:17.587431 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 874) took 1991 ms. Trace:
I20251024 19:48:17.587791 1776 rpcz_store.cc:276] 1024 19:48:15.595575 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.598861 (+ 3286us) service_pool.cc:225] Handling call
1024 19:48:17.587327 (+1988466us) ts_tablet_manager.cc:1151] Deleting tablet 63518b583ac94fa6abde420e176ec8ba
1024 19:48:17.587337 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.587413 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1987269}
W20251024 19:48:17.588835 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 452f77091adc4cf09de2231f2f6d845b because the tablet was not found. No further retry: Not found: Tablet not found: 452f77091adc4cf09de2231f2f6d845b
W20251024 19:48:17.589262 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2cbeaac19e834ecf98e5b8afc952b9da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cbeaac19e834ecf98e5b8afc952b9da already in progress: creating tablet
W20251024 19:48:17.589708 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 63518b583ac94fa6abde420e176ec8ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63518b583ac94fa6abde420e176ec8ba already in progress: creating tablet
I20251024 19:48:17.588500 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 876) took 1894 ms. Trace:
I20251024 19:48:17.589335 1612 tablet_service.cc:1505] Processing CreateTablet for tablet ef837960a0544202834a0ca509089b29 (DEFAULT_TABLE table=default.Table_f76a1631002d4ee0a3323bcbd2e339d3 [id=1dca2c64837343e69b1f8c12747a68d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.590487 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef837960a0544202834a0ca509089b29. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:17.591070 1447 consensus_peers.cc:597] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:17.589406 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 6a0027505cee405cb3a6feb17e08068e (DEFAULT_TABLE table=default.Table_b841d8eb9cca40dcb3e45759fbf5acce [id=8b138d8aaace4167bc8ce84c993cde94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.594183 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a0027505cee405cb3a6feb17e08068e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.589781 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 44305f9f044b4b14bbf66bbe612d14a1 (DEFAULT_TABLE table=default.Table_039c8e8c0a8946a59f10e06691e5b946 [id=90ad525544aa4ba4a212ff2e5ec3ba32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.598052 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44305f9f044b4b14bbf66bbe612d14a1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:17.601310 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46a740e6ac1b49138a5a3523961107f1 consensus state has no leader
I20251024 19:48:17.601610 1775 ts_tablet_manager.cc:1929] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:17.601724 1775 log.cc:1199] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/7c77f6bbf7b145389ca6cf74ddd3aa3f
I20251024 19:48:17.602422 1768 tablet_bootstrap.cc:654] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.605249 2936 raft_consensus.cc:493] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.605381 2936 raft_consensus.cc:515] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.606009 2936 leader_election.cc:290] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:17.606165 1774 ts_tablet_manager.cc:1929] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:17.606261 1774 log.cc:1199] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/1d04a9b54bd6446d8deb64292cef672d
I20251024 19:48:17.606832 1774 ts_tablet_manager.cc:1950] T 1d04a9b54bd6446d8deb64292cef672d P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:17.606925 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e29abfa3bc8944aea7d1acd03e3eb897" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:17.607374 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 717) took 5017 ms. Trace:
I20251024 19:48:17.607473 1774 rpcz_store.cc:276] 1024 19:48:12.589667 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.591380 (+ 1713us) service_pool.cc:225] Handling call
1024 19:48:17.077305 (+4485925us) ts_tablet_manager.cc:1151] Deleting tablet 1d04a9b54bd6446d8deb64292cef672d
1024 19:48:17.077313 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.462696 (+385383us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.606076 (+143380us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.607330 (+ 1254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4485759}
I20251024 19:48:17.608152 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 724) took 4488 ms. Trace:
I20251024 19:48:17.608290 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e29abfa3bc8944aea7d1acd03e3eb897" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:17.608275 1774 rpcz_store.cc:276] 1024 19:48:13.119537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.120410 (+ 873us) service_pool.cc:225] Handling call
1024 19:48:17.607776 (+4487366us) ts_tablet_manager.cc:1151] Deleting tablet 64325d89db714554866e74922d103e34
1024 19:48:17.607786 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.608118 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4434051}
I20251024 19:48:17.608749 1774 tablet_replica.cc:333] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:17.608939 1766 tablet_bootstrap.cc:492] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:17.609087 1766 ts_tablet_manager.cc:1403] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.114s user 0.001s sys 0.003s
W20251024 19:48:17.609372 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 346cb0bb8a0d43f7a15b788a303a6bcf consensus state has no leader
I20251024 19:48:17.609916 1766 raft_consensus.cc:359] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.610080 1766 raft_consensus.cc:385] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.610131 1766 raft_consensus.cc:740] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.610322 1766 consensus_queue.cc:260] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.610661 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet 1295c99836d14a1cb0002e6c6aaee1eb from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:17.611689 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d2f10524d22141078abbeeea11a129ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:17.613085 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2f10524d22141078abbeeea11a129ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.613368 1766 ts_tablet_manager.cc:1434] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251024 19:48:17.613684 1766 tablet_bootstrap.cc:492] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:17.615008 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_33ccbd1889d74b2e909e6ede2f837ef1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:17.616218 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33ccbd1889d74b2e909e6ede2f837ef1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.616917 1766 tablet_bootstrap.cc:654] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.617720 1767 tablet_bootstrap.cc:492] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:17.617928 1767 ts_tablet_manager.cc:1403] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.000s
I20251024 19:48:17.618813 1767 raft_consensus.cc:359] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.619060 1767 raft_consensus.cc:385] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.619145 1767 raft_consensus.cc:740] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.619386 1767 consensus_queue.cc:260] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.619598 1775 ts_tablet_manager.cc:1950] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:17.619884 2920 raft_consensus.cc:493] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.620620 2920 raft_consensus.cc:515] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.621357 2920 leader_election.cc:290] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.622435 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9489a01434a24d9ca94791252950725c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:17.622675 1507 raft_consensus.cc:2468] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:17.623147 1586 leader_election.cc:304] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:17.623473 2920 raft_consensus.cc:2804] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:17.623540 2920 raft_consensus.cc:493] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:17.623602 2920 raft_consensus.cc:3060] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.626598 2920 raft_consensus.cc:515] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.627771 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9489a01434a24d9ca94791252950725c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:17.627971 1507 raft_consensus.cc:3060] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.630185 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 813) took 3340 ms. Trace:
I20251024 19:48:17.630273 1775 rpcz_store.cc:276] 1024 19:48:14.289293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.289358 (+ 65us) service_pool.cc:225] Handling call
1024 19:48:17.452557 (+3163199us) ts_tablet_manager.cc:1151] Deleting tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f
1024 19:48:17.452568 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.465482 (+ 12914us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x61600005b354. stack: 00007f26c12049f6 00007f26c12047b0 00007f26c01e03ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f26cd0ca843 00007f26ce417ae3 00007f26d0fd4287 00007f26d10072d4 00007f26d0fe7d70 00007f26d0fe7afd 0000000000a8867c 00007f26c124ad7f 00007f26c124e04b 00007f26c124dec8
1024 19:48:17.531637 (+ 66155us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.601487 (+ 69850us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.630157 (+ 28670us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8567040,"tablet-delete.queue_time_us":3162971}
I20251024 19:48:17.630828 1775 tablet_replica.cc:333] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:17.630882 1507 raft_consensus.cc:2468] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:17.631035 1775 raft_consensus.cc:2243] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:17.631386 1775 raft_consensus.cc:2272] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.631989 1775 ts_tablet_manager.cc:1916] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.632099 1586 leader_election.cc:304] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:17.632370 2569 raft_consensus.cc:2804] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:17.632479 2569 raft_consensus.cc:697] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:17.632719 2569 consensus_queue.cc:237] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.637256 2920 leader_election.cc:290] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.641819 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_bd37f8f763d147b2b7656b63f72fd749" } modify_external_catalogs: true
I20251024 19:48:17.642204 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_bd37f8f763d147b2b7656b63f72fd749" } modify_external_catalogs: true
I20251024 19:48:17.642500 1767 ts_tablet_manager.cc:1434] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20251024 19:48:17.643009 1767 tablet_bootstrap.cc:492] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:17.645783 1767 tablet_bootstrap.cc:654] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.648885 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c7859afca8534349935391ebab856798" } modify_external_catalogs: true
I20251024 19:48:17.643187 1776 rpcz_store.cc:276] 1024 19:48:15.693951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.694033 (+ 82us) service_pool.cc:225] Handling call
1024 19:48:17.588398 (+1894365us) ts_tablet_manager.cc:1151] Deleting tablet 605205170c994296a5ef0dc568788517
1024 19:48:17.588407 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.588485 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1891438}
I20251024 19:48:17.649190 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c7859afca8534349935391ebab856798" } modify_external_catalogs: true
I20251024 19:48:17.649688 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 875) took 1957 ms. Trace:
I20251024 19:48:17.649832 1776 rpcz_store.cc:276] 1024 19:48:15.692130 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.694264 (+ 2134us) service_pool.cc:225] Handling call
1024 19:48:17.649477 (+1955213us) ts_tablet_manager.cc:1151] Deleting tablet bf241a9566c34b7b911b69aaf4de06a2
1024 19:48:17.649489 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.649663 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1952297}
I20251024 19:48:17.650408 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 890) took 1073 ms. Trace:
I20251024 19:48:17.650553 1776 rpcz_store.cc:276] 1024 19:48:16.577257 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.577311 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:17.650290 (+1072979us) ts_tablet_manager.cc:1151] Deleting tablet 265040336fb347cca1d52dd331a1418d
1024 19:48:17.650301 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.650390 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045800}
W20251024 19:48:17.650753 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 605205170c994296a5ef0dc568788517 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 605205170c994296a5ef0dc568788517 already in progress: creating tablet
W20251024 19:48:17.651074 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet bf241a9566c34b7b911b69aaf4de06a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf241a9566c34b7b911b69aaf4de06a2 already in progress: creating tablet
I20251024 19:48:17.651113 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 891) took 1072 ms. Trace:
I20251024 19:48:17.651245 1776 rpcz_store.cc:276] 1024 19:48:16.579128 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.579177 (+ 49us) service_pool.cc:225] Handling call
1024 19:48:17.650999 (+1071822us) ts_tablet_manager.cc:1151] Deleting tablet 2e6ab4e8ffb84bf0961af481f0d4b180
1024 19:48:17.651010 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.651096 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046427}
W20251024 19:48:17.651757 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 265040336fb347cca1d52dd331a1418d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 265040336fb347cca1d52dd331a1418d already in progress: creating tablet
W20251024 19:48:17.652077 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2e6ab4e8ffb84bf0961af481f0d4b180 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e6ab4e8ffb84bf0961af481f0d4b180 already in progress: creating tablet
I20251024 19:48:17.653136 2923 consensus_queue.cc:1048] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:17.654031 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 892) took 1019 ms. Trace:
I20251024 19:48:17.654182 1776 rpcz_store.cc:276] 1024 19:48:16.634217 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.634271 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:17.651670 (+1017399us) ts_tablet_manager.cc:1151] Deleting tablet 9026d9e801294dd2bfdb9c90e45a0990
1024 19:48:17.651681 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.654013 (+ 2332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1017189}
I20251024 19:48:17.654368 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_809ec8b3a0fc4fa08457269a52eae680" } new_table_name: "default.Table_9b365a44719d40d8a8b425b94fb089e3" modify_external_catalogs: true
I20251024 19:48:17.655017 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 893) took 1010 ms. Trace:
I20251024 19:48:17.655160 1776 rpcz_store.cc:276] 1024 19:48:16.645016 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.645087 (+ 71us) service_pool.cc:225] Handling call
1024 19:48:17.654673 (+1009586us) ts_tablet_manager.cc:1151] Deleting tablet 41176885447e42409ee4363bb4d903f2
1024 19:48:17.654920 (+ 247us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.655000 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1009362}
I20251024 19:48:17.655290 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_88a506f2737145189da07becb2912c29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:17.655911 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88a506f2737145189da07becb2912c29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.656638 1776 tablet_replica.cc:333] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:17.656867 1776 raft_consensus.cc:2243] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:17.657018 1776 raft_consensus.cc:2272] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.657001 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e739c763ff31475a9a277870eb5d0132"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:17.657545 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e739c763ff31475a9a277870eb5d0132 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.657616 1776 ts_tablet_manager.cc:1916] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.661300 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e81edbfed6c14c7cb43ba38ef86e78ee" } modify_external_catalogs: true
I20251024 19:48:17.661610 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e81edbfed6c14c7cb43ba38ef86e78ee" } modify_external_catalogs: true
W20251024 19:48:17.663084 1178 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 46a740e6ac1b49138a5a3523961107f1 (table default.Table_71777a1b569c44559975f3db7d8cf059 [id=198216eca4fd430aa72fbffa70da3bdf], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 46a740e6ac1b49138a5a3523961107f1 consensus state has no leader
W20251024 19:48:17.663502 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5688b62d37d6453987bb4088c849986b consensus state has no leader
I20251024 19:48:17.664462 1767 tablet_bootstrap.cc:492] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:17.664462 1766 tablet_bootstrap.cc:492] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:17.664621 1766 ts_tablet_manager.cc:1403] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20251024 19:48:17.665254 1767 ts_tablet_manager.cc:1403] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20251024 19:48:17.666018 1766 raft_consensus.cc:359] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.666222 1766 raft_consensus.cc:385] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.665829 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b6803d23a4714934b85190886611d8d1
I20251024 19:48:17.666285 1766 raft_consensus.cc:740] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.666568 1766 consensus_queue.cc:260] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.667140 1766 ts_tablet_manager.cc:1434] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251024 19:48:17.666929 1767 raft_consensus.cc:359] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.667716 1767 raft_consensus.cc:385] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.667889 1767 raft_consensus.cc:740] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.668196 1766 tablet_bootstrap.cc:492] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:17.667989 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9026d9e801294dd2bfdb9c90e45a0990 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9026d9e801294dd2bfdb9c90e45a0990 already in progress: creating tablet
W20251024 19:48:17.668406 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 41176885447e42409ee4363bb4d903f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41176885447e42409ee4363bb4d903f2 already in progress: creating tablet
I20251024 19:48:17.668196 1767 consensus_queue.cc:260] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:17.668920 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 27b32126d04e480f8db7b2b20aba5873 because the tablet was not found. No further retry: Not found: Tablet not found: 27b32126d04e480f8db7b2b20aba5873
I20251024 19:48:17.669301 2857 raft_consensus.cc:493] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.669433 2857 raft_consensus.cc:515] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:17.669591 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet cef80e4d14bb4e719bceebf43041d079 because the tablet was not found. No further retry: Not found: Tablet not found: cef80e4d14bb4e719bceebf43041d079
I20251024 19:48:17.669536 1767 ts_tablet_manager.cc:1434] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:17.670094 2857 leader_election.cc:290] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.670127 1767 tablet_bootstrap.cc:492] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:17.670693 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129debb55f4a4b7295a8ad88e55ebc83" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:17.671078 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet b6803d23a4714934b85190886611d8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:17.671353 1444 leader_election.cc:343] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.674858 1767 tablet_bootstrap.cc:654] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.676414 2920 raft_consensus.cc:493] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.677922 2920 raft_consensus.cc:515] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.678581 2920 leader_election.cc:290] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.678694 1766 tablet_bootstrap.cc:654] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.679791 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f89a3bbeb9b4d57a49078729110d951" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:17.679993 1507 raft_consensus.cc:2468] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:17.680959 1586 leader_election.cc:304] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:17.681351 2920 raft_consensus.cc:2804] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:17.681460 2920 raft_consensus.cc:493] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:17.681535 2920 raft_consensus.cc:3060] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.684367 1775 ts_tablet_manager.cc:1929] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:17.684301 2920 raft_consensus.cc:515] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.684471 1775 log.cc:1199] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/505e4fb0755a4caaa462e5f5154bfee2
I20251024 19:48:17.685055 1775 ts_tablet_manager.cc:1950] T 505e4fb0755a4caaa462e5f5154bfee2 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:17.685150 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2c96c11697514af6bf4e80a67aadf748
I20251024 19:48:17.685639 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f89a3bbeb9b4d57a49078729110d951" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:17.685803 1507 raft_consensus.cc:3060] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.688589 1507 raft_consensus.cc:2468] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:17.689203 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c96c11697514af6bf4e80a67aadf748 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.689805 1586 leader_election.cc:304] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:17.690142 2923 raft_consensus.cc:2804] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:17.690277 2923 raft_consensus.cc:697] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:17.690582 2923 consensus_queue.cc:237] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.692970 2920 leader_election.cc:290] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.698385 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 332d8bd38f0b478d9592c781dfd5ca72 (DEFAULT_TABLE table=default.Table_3864c444fa71452eb70094031c884901 [id=6c7a3a1c60c44263b5b29202cf45ad69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.698599 1766 tablet_bootstrap.cc:492] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:17.698745 1766 ts_tablet_manager.cc:1403] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20251024 19:48:17.698844 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332d8bd38f0b478d9592c781dfd5ca72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.699496 1766 raft_consensus.cc:359] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.699635 1766 raft_consensus.cc:385] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.699687 1766 raft_consensus.cc:740] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.699854 1766 consensus_queue.cc:260] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.700278 1766 ts_tablet_manager.cc:1434] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251024 19:48:17.701012 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3c3587f8ff21469a956f564c794672c5" } modify_external_catalogs: true
I20251024 19:48:17.701390 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3c3587f8ff21469a956f564c794672c5" } modify_external_catalogs: true
I20251024 19:48:17.702284 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 812) took 3413 ms. Trace:
I20251024 19:48:17.702677 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 404c38f8db2f43bfbc63e9ba2cee902c (DEFAULT_TABLE table=default.Table_f1b82a4bcb2d4cca842b1be702afda63 [id=bb53c691d85f471bb66c71780f9c93e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.703150 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 404c38f8db2f43bfbc63e9ba2cee902c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.704663 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9c8900b6f68e44cbb3333b6ffcb23eac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:17.702369 1775 rpcz_store.cc:276] 1024 19:48:14.289174 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.289654 (+ 480us) service_pool.cc:225] Handling call
1024 19:48:17.630769 (+3341115us) ts_tablet_manager.cc:1151] Deleting tablet 505e4fb0755a4caaa462e5f5154bfee2
1024 19:48:17.630785 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.643648 (+ 12863us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.684244 (+ 40596us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.702262 (+ 18018us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3340790}
W20251024 19:48:17.705222 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c8900b6f68e44cbb3333b6ffcb23eac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.705940 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 814) took 3409 ms. Trace:
I20251024 19:48:17.706018 1775 rpcz_store.cc:276] 1024 19:48:14.296372 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.296812 (+ 440us) service_pool.cc:225] Handling call
1024 19:48:17.705523 (+3408711us) ts_tablet_manager.cc:1151] Deleting tablet cf18f375266b4448b3f67afd787df6ad
1024 19:48:17.705532 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.705919 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3408513}
I20251024 19:48:17.707293 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 815) took 3394 ms. Trace:
I20251024 19:48:17.707581 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 19d83464fbf44f78881c7cae43a72091 (DEFAULT_TABLE table=default.Table_da7aeba16fdd4a0fa3d17ada53c2f9fb [id=a9136e43a07b4d868d017534d3472d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.707937 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19d83464fbf44f78881c7cae43a72091. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.708500 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 2d8b8a1cff3540f7b7f13712425d534e (DEFAULT_TABLE table=default.Table_761f1d6d97dc46c28951c41e9c5082b3 [id=767e92e851544b10a18bd5d5de23d9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.708833 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d8b8a1cff3540f7b7f13712425d534e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.713045 1775 rpcz_store.cc:276] 1024 19:48:14.312363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.312516 (+ 153us) service_pool.cc:225] Handling call
1024 19:48:17.707182 (+3394666us) ts_tablet_manager.cc:1151] Deleting tablet 839cef50795840fab564b94d88565f8c
1024 19:48:17.707191 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.707273 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3394469}
I20251024 19:48:17.712894 1766 tablet_bootstrap.cc:492] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:17.715111 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_77d2dbff62c647e2b909bc059207d895"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:17.715835 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77d2dbff62c647e2b909bc059207d895 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.716101 1766 tablet_bootstrap.cc:654] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.718007 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 818) took 3158 ms. Trace:
I20251024 19:48:17.718678 1775 rpcz_store.cc:276] 1024 19:48:14.559887 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.565665 (+ 5778us) service_pool.cc:225] Handling call
1024 19:48:17.717905 (+3152240us) ts_tablet_manager.cc:1151] Deleting tablet fd52295b14b14e70bbdcb45f1bd849e4
1024 19:48:17.717916 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.717989 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3152013}
I20251024 19:48:17.719558 1775 tablet_replica.cc:333] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:17.719580 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b724eadbb86e4eeab4ba8dfd565290ff
I20251024 19:48:17.720088 1775 raft_consensus.cc:2243] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:17.720450 1775 raft_consensus.cc:2272] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.721042 1775 ts_tablet_manager.cc:1916] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.721364 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_468b839284304d4eb9920feeeb49bc18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:17.722075 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_468b839284304d4eb9920feeeb49bc18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.724447 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet b724eadbb86e4eeab4ba8dfd565290ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.724838 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f4e812c43ade4831af50825474976c8e
I20251024 19:48:17.730793 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet f4e812c43ade4831af50825474976c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:17.737326 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fe56cf0299064493aa6b4f993e0ad929"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:17.738206 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe56cf0299064493aa6b4f993e0ad929 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:17.739640 3077 consensus_queue.cc:1048] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251024 19:48:17.740918 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_df275b460e8245c8bc231bab1fb44630" } new_table_name: "default.Table_cc788c9fa46e4cdb9fcfe197121d4253" modify_external_catalogs: true
I20251024 19:48:17.608805 1305 leader_election.cc:304] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.621371 1768 tablet_bootstrap.cc:492] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
W20251024 19:48:17.743914 1305 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 135082us
I20251024 19:48:17.744025 1768 ts_tablet_manager.cc:1403] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.266s user 0.012s sys 0.008s
I20251024 19:48:17.744704 2936 raft_consensus.cc:2749] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.744870 1768 raft_consensus.cc:359] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.745090 1768 raft_consensus.cc:385] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.745175 1768 raft_consensus.cc:740] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.745265 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 94fc8c5716af47faa4e831f05fce8610 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.745334 1365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129debb55f4a4b7295a8ad88e55ebc83" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:17.745396 1768 consensus_queue.cc:260] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.745831 1768 ts_tablet_manager.cc:1434] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:17.745947 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 6a0027505cee405cb3a6feb17e08068e (DEFAULT_TABLE table=default.Table_b841d8eb9cca40dcb3e45759fbf5acce [id=8b138d8aaace4167bc8ce84c993cde94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.746157 1768 tablet_bootstrap.cc:492] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:17.623450 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9489a01434a24d9ca94791252950725c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:17.637055 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9489a01434a24d9ca94791252950725c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:17.680392 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f89a3bbeb9b4d57a49078729110d951" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:17.747373 1586 leader_election.cc:343] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.692579 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f89a3bbeb9b4d57a49078729110d951" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:17.747747 1586 leader_election.cc:343] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:17.748032 1586 leader_election.cc:343] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:17.748283 1586 leader_election.cc:343] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.752452 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet aeeaf256b1404653b690c712d9ef7a6d from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:17.752912 2920 raft_consensus.cc:493] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.753155 2920 raft_consensus.cc:515] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.754174 1768 tablet_bootstrap.cc:654] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.754998 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "844638a6b6944d0394df3e78a8ea51d6" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:17.755213 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 94fc8c5716af47faa4e831f05fce8610 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.755244 1507 raft_consensus.cc:2468] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:17.755790 1586 leader_election.cc:304] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:17.756670 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 404c38f8db2f43bfbc63e9ba2cee902c (DEFAULT_TABLE table=default.Table_f1b82a4bcb2d4cca842b1be702afda63 [id=bb53c691d85f471bb66c71780f9c93e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.756755 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 19d83464fbf44f78881c7cae43a72091 (DEFAULT_TABLE table=default.Table_da7aeba16fdd4a0fa3d17ada53c2f9fb [id=a9136e43a07b4d868d017534d3472d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.756830 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet b6803d23a4714934b85190886611d8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.757068 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c96c11697514af6bf4e80a67aadf748 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.757139 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 2d8b8a1cff3540f7b7f13712425d534e (DEFAULT_TABLE table=default.Table_761f1d6d97dc46c28951c41e9c5082b3 [id=767e92e851544b10a18bd5d5de23d9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.757221 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet b724eadbb86e4eeab4ba8dfd565290ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.757346 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 332d8bd38f0b478d9592c781dfd5ca72 (DEFAULT_TABLE table=default.Table_3864c444fa71452eb70094031c884901 [id=6c7a3a1c60c44263b5b29202cf45ad69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.757617 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 44305f9f044b4b14bbf66bbe612d14a1 (DEFAULT_TABLE table=default.Table_039c8e8c0a8946a59f10e06691e5b946 [id=90ad525544aa4ba4a212ff2e5ec3ba32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.757961 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 6a0027505cee405cb3a6feb17e08068e (DEFAULT_TABLE table=default.Table_b841d8eb9cca40dcb3e45759fbf5acce [id=8b138d8aaace4167bc8ce84c993cde94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.758246 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "844638a6b6944d0394df3e78a8ea51d6" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:17.758608 1339 tablet_service.cc:1505] Processing CreateTablet for tablet ef837960a0544202834a0ca509089b29 (DEFAULT_TABLE table=default.Table_f76a1631002d4ee0a3323bcbd2e339d3 [id=1dca2c64837343e69b1f8c12747a68d1]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:17.760349 1447 leader_election.cc:343] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.760485 1447 leader_election.cc:304] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.760736 2857 raft_consensus.cc:2749] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.760949 2920 leader_election.cc:290] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.761150 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet b6803d23a4714934b85190886611d8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.761355 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c96c11697514af6bf4e80a67aadf748 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.761534 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 44305f9f044b4b14bbf66bbe612d14a1 (DEFAULT_TABLE table=default.Table_039c8e8c0a8946a59f10e06691e5b946 [id=90ad525544aa4ba4a212ff2e5ec3ba32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.761669 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 404c38f8db2f43bfbc63e9ba2cee902c (DEFAULT_TABLE table=default.Table_f1b82a4bcb2d4cca842b1be702afda63 [id=bb53c691d85f471bb66c71780f9c93e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.761915 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44305f9f044b4b14bbf66bbe612d14a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.762089 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 332d8bd38f0b478d9592c781dfd5ca72 (DEFAULT_TABLE table=default.Table_3864c444fa71452eb70094031c884901 [id=6c7a3a1c60c44263b5b29202cf45ad69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.762180 1477 tablet_service.cc:1505] Processing CreateTablet for tablet ef837960a0544202834a0ca509089b29 (DEFAULT_TABLE table=default.Table_f76a1631002d4ee0a3323bcbd2e339d3 [id=1dca2c64837343e69b1f8c12747a68d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.762392 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332d8bd38f0b478d9592c781dfd5ca72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.762635 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef837960a0544202834a0ca509089b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.762805 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 404c38f8db2f43bfbc63e9ba2cee902c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.763873 2857 raft_consensus.cc:493] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.763988 2857 raft_consensus.cc:515] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.764541 2857 leader_election.cc:290] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.765164 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf4596b333244cc9e9e35a128721e4a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:17.765828 1480 tablet_service.cc:1505] Processing CreateTablet for tablet 2d8b8a1cff3540f7b7f13712425d534e (DEFAULT_TABLE table=default.Table_761f1d6d97dc46c28951c41e9c5082b3 [id=767e92e851544b10a18bd5d5de23d9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.765925 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 19d83464fbf44f78881c7cae43a72091 (DEFAULT_TABLE table=default.Table_da7aeba16fdd4a0fa3d17ada53c2f9fb [id=a9136e43a07b4d868d017534d3472d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.766112 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d8b8a1cff3540f7b7f13712425d534e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:17.766187 1444 leader_election.cc:343] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.766259 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19d83464fbf44f78881c7cae43a72091. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.761894 2920 raft_consensus.cc:2804] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:17.766448 2920 raft_consensus.cc:493] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:17.766625 2920 raft_consensus.cc:3060] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.769886 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a0027505cee405cb3a6feb17e08068e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.776340 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf4596b333244cc9e9e35a128721e4a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:17.780009 1447 leader_election.cc:343] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.780167 1447 leader_election.cc:304] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.780448 2857 raft_consensus.cc:2749] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.784869 2920 raft_consensus.cc:515] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.796553 2857 raft_consensus.cc:493] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.796741 2857 raft_consensus.cc:515] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.798152 2858 consensus_queue.cc:1048] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:17.798460 1774 raft_consensus.cc:2243] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:17.798879 1774 raft_consensus.cc:2272] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.799333 2857 leader_election.cc:290] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.799464 2857 raft_consensus.cc:493] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.799559 2857 raft_consensus.cc:515] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.800056 1774 ts_tablet_manager.cc:1916] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.800020 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c92b080805845c6af8f1cb160110915" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:17.800132 2857 leader_election.cc:290] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:17.800488 1447 leader_election.cc:343] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.800839 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94fc8c5716af47faa4e831f05fce8610" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:17.801178 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c92b080805845c6af8f1cb160110915" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:17.801637 1447 leader_election.cc:343] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.803272 2920 leader_election.cc:290] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.806573 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 404c38f8db2f43bfbc63e9ba2cee902c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.808182 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94fc8c5716af47faa4e831f05fce8610" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:17.808583 1444 leader_election.cc:343] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.808701 1444 leader_election.cc:304] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:17.809331 1444 leader_election.cc:343] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.809427 1444 leader_election.cc:304] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.809820 2857 raft_consensus.cc:2749] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.809978 2857 raft_consensus.cc:2749] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:17.810942 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60fbfb1620934034af2acaa39677554f consensus state has no leader
I20251024 19:48:17.817914 1775 ts_tablet_manager.cc:1929] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:17.821913 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet b724eadbb86e4eeab4ba8dfd565290ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.823485 1775 log.cc:1199] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/187bc968c2b049028dd25dfb0264c0aa
I20251024 19:48:17.825505 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet f4e812c43ade4831af50825474976c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:17.826077 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet f4e812c43ade4831af50825474976c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:17.826253 2858 raft_consensus.cc:493] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.826391 2858 raft_consensus.cc:515] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.827087 2858 leader_election.cc:290] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.827503 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6db58fac5e4af9b27bde6c223aa845" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:17.827843 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6db58fac5e4af9b27bde6c223aa845" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:17.828166 1444 leader_election.cc:343] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:17.828351 1447 leader_election.cc:343] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.828467 1447 leader_election.cc:304] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.828830 2858 raft_consensus.cc:2749] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.830657 1776 ts_tablet_manager.cc:1929] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:17.833997 1768 tablet_bootstrap.cc:492] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:17.834262 1776 log.cc:1199] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/f51c56337ebb450aaf62dcc81ef63ae8
I20251024 19:48:17.834414 1768 ts_tablet_manager.cc:1403] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.088s user 0.009s sys 0.000s
I20251024 19:48:17.834892 1776 ts_tablet_manager.cc:1950] T f51c56337ebb450aaf62dcc81ef63ae8 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:17.834095 2920 raft_consensus.cc:493] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.836371 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet f51c56337ebb450aaf62dcc81ef63ae8 (table default.Table_efcd2af1913c4a73ba5148272aa880aa [id=01a8150e819b4075810a95720eaa5840]) successfully deleted
I20251024 19:48:17.836369 2920 raft_consensus.cc:515] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.836305 1768 raft_consensus.cc:359] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.836746 1768 raft_consensus.cc:385] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.836948 1768 raft_consensus.cc:740] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.837307 2920 leader_election.cc:290] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.837527 1768 consensus_queue.cc:260] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.839020 1776 tablet_replica.cc:333] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:17.839380 1776 raft_consensus.cc:2243] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:17.840178 1776 raft_consensus.cc:2272] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.826282 1775 ts_tablet_manager.cc:1950] T 187bc968c2b049028dd25dfb0264c0aa P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:17.846580 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet aeeaf256b1404653b690c712d9ef7a6d
I20251024 19:48:17.850693 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet aeeaf256b1404653b690c712d9ef7a6d with delete_type TABLET_DATA_DELETED (replaced by tablet 6a93c672295a40e5a79501c0ff3d1a3c) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:17.853876 3065 log.cc:927] Time spent T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20251024 19:48:17.858139 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 820) took 3298 ms. Trace:
W20251024 19:48:17.858577 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 102823us
I20251024 19:48:17.862982 1186 catalog_manager.cc:6719] ReplaceTablet: tablet aeeaf256b1404653b690c712d9ef7a6d deleted and replaced by tablet 6a93c672295a40e5a79501c0ff3d1a3c
I20251024 19:48:17.864826 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "844638a6b6944d0394df3e78a8ea51d6" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:17.864768 1768 ts_tablet_manager.cc:1434] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20251024 19:48:17.865746 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "844638a6b6944d0394df3e78a8ea51d6" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:17.860208 1776 ts_tablet_manager.cc:1916] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.865969 1507 raft_consensus.cc:3060] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.860306 2923 raft_consensus.cc:493] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.866127 2923 raft_consensus.cc:515] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.866871 2923 leader_election.cc:290] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.867064 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 03d8ab62b47f474198841ea14a3b7b00 (DEFAULT_TABLE table=default.Table_1312dfdab750454a8f412ce6a66284ac [id=28dfd9c9c5cf470abbcfad936688bf94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.867456 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03d8ab62b47f474198841ea14a3b7b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.869081 1507 raft_consensus.cc:2468] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:17.864946 1775 rpcz_store.cc:276] 1024 19:48:14.560119 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.569622 (+ 9503us) service_pool.cc:225] Handling call
1024 19:48:17.719506 (+3149884us) ts_tablet_manager.cc:1151] Deleting tablet 187bc968c2b049028dd25dfb0264c0aa
1024 19:48:17.719517 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.792603 (+ 73086us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.817788 (+ 25185us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.858113 (+ 40325us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14836,"tablet-delete.queue_time_us":3149682}
I20251024 19:48:17.873898 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16bc737104ab46b3b071a0760dbc847d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:17.874317 1507 raft_consensus.cc:2468] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:17.874904 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 825) took 3216 ms. Trace:
I20251024 19:48:17.878131 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16bc737104ab46b3b071a0760dbc847d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:17.878659 1768 tablet_bootstrap.cc:492] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:17.879570 1586 leader_election.cc:343] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.858676 1766 tablet_bootstrap.cc:492] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:17.879920 1775 rpcz_store.cc:276] 1024 19:48:14.658359 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.658477 (+ 118us) service_pool.cc:225] Handling call
1024 19:48:17.874781 (+3216304us) ts_tablet_manager.cc:1151] Deleting tablet b3e9960b7821421aa23e7ef5389f894a
1024 19:48:17.874792 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.874889 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3216090}
I20251024 19:48:17.880059 1766 ts_tablet_manager.cc:1403] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.167s user 0.008s sys 0.003s
I20251024 19:48:17.881183 1766 raft_consensus.cc:359] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:17.881412 1586 leader_election.cc:343] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:17.881680 1586 consensus_peers.cc:597] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:17.881851 1586 leader_election.cc:343] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.882647 1766 raft_consensus.cc:385] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.882877 1766 raft_consensus.cc:740] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.883314 1766 consensus_queue.cc:260] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.886165 1766 ts_tablet_manager.cc:1434] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251024 19:48:17.887651 1766 tablet_bootstrap.cc:492] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:17.889303 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 828) took 3201 ms. Trace:
I20251024 19:48:17.890393 1775 rpcz_store.cc:276] 1024 19:48:14.687707 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.687798 (+ 91us) service_pool.cc:225] Handling call
1024 19:48:17.880231 (+3192433us) ts_tablet_manager.cc:1151] Deleting tablet e51e597b8fd044e393a3af07da25586c
1024 19:48:17.880246 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.889281 (+ 9035us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3192035}
I20251024 19:48:17.891005 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 829) took 3200 ms. Trace:
I20251024 19:48:17.891335 1775 rpcz_store.cc:276] 1024 19:48:14.690195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.690929 (+ 734us) service_pool.cc:225] Handling call
1024 19:48:17.890892 (+3199963us) ts_tablet_manager.cc:1151] Deleting tablet 9684072368f14cc2942b4e240f90b0f2
1024 19:48:17.890901 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.890991 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3199521}
I20251024 19:48:17.892056 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 830) took 3173 ms. Trace:
I20251024 19:48:17.892184 1775 rpcz_store.cc:276] 1024 19:48:14.718817 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.718869 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:17.891938 (+3173069us) ts_tablet_manager.cc:1151] Deleting tablet 7cab9c17496947efbbe3cd9e5d8a49a8
1024 19:48:17.891947 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.892040 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3143076}
I20251024 19:48:17.892673 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 831) took 3170 ms. Trace:
I20251024 19:48:17.892791 1775 rpcz_store.cc:276] 1024 19:48:14.721854 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.725393 (+ 3539us) service_pool.cc:225] Handling call
1024 19:48:17.892562 (+3167169us) ts_tablet_manager.cc:1151] Deleting tablet 292c31e7d4ce411bb336a1405ca43ba5
1024 19:48:17.892570 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.892649 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3143553}
I20251024 19:48:17.893250 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 833) took 3107 ms. Trace:
I20251024 19:48:17.893371 1775 rpcz_store.cc:276] 1024 19:48:14.785421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.785947 (+ 526us) service_pool.cc:225] Handling call
1024 19:48:17.893140 (+3107193us) ts_tablet_manager.cc:1151] Deleting tablet 520060a9adca4bb0b8616bcc006dce52
1024 19:48:17.893148 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.893236 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3106931}
I20251024 19:48:17.894343 1586 leader_election.cc:304] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:17.889537 1766 tablet_bootstrap.cc:654] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.894563 2936 raft_consensus.cc:493] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.894718 2936 raft_consensus.cc:515] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.895298 2936 leader_election.cc:290] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:17.895787 2923 raft_consensus.cc:2804] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:17.895928 2923 raft_consensus.cc:697] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:17.896044 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 1d04a9b54bd6446d8deb64292cef672d (table default.Table_147e888401aa40e5a60d97884b36cb98 [id=1acdc5d0bfb94e0faf47055b20d803b0]) successfully deleted
I20251024 19:48:17.896166 2923 consensus_queue.cc:237] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:17.896315 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 64325d89db714554866e74922d103e34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64325d89db714554866e74922d103e34 already in progress: creating tablet
W20251024 19:48:17.896452 1308 leader_election.cc:343] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 187bc968c2b049028dd25dfb0264c0aa
I20251024 19:48:17.896514 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f (table default.Table_a1d09325af844d71b290d95b8a4aaf12 [id=b62ec7bcad7c41b1b946e9ebf2f4bf39]) successfully deleted
I20251024 19:48:17.896728 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 505e4fb0755a4caaa462e5f5154bfee2 (table default.Table_74e4f24f12db403686548924c83c5191 [id=508d6cdc5d4a494bad31eaf36383e214]) successfully deleted
I20251024 19:48:17.900324 1586 leader_election.cc:304] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:17.901785 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 346cb0bb8a0d43f7a15b788a303a6bcf (DEFAULT_TABLE table=default.Table_532ecd1242ea40d29a9d1bcf6fa8a488 [id=8a0022a43f6a4a46b3a0dfd1942b5e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.902081 3080 raft_consensus.cc:2804] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:17.902153 3080 raft_consensus.cc:493] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:17.902191 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 346cb0bb8a0d43f7a15b788a303a6bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.902215 3080 raft_consensus.cc:3060] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.904994 3080 raft_consensus.cc:515] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.905856 3080 leader_election.cc:290] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:17.906174 1305 leader_election.cc:343] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 187bc968c2b049028dd25dfb0264c0aa
I20251024 19:48:17.906298 1305 leader_election.cc:304] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.906824 2936 raft_consensus.cc:2749] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.907033 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2602335327fc489fa154056c5ef5138f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:17.908479 1366 raft_consensus.cc:3060] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.899890 1187 catalog_manager.cc:5649] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:17.911480 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 9d61e01cb1c240a0b4edd77621a86d75 (DEFAULT_TABLE table=default.Table_6011e1b8d3634908b57b079b4dea405e [id=782290308c2048ca81343ec18770b3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.911854 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d61e01cb1c240a0b4edd77621a86d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.911841 1187 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 81a3de69a13c459b9b7c02761ab7261e (table default.Table_b5a66c4e9cd8408782690b66cdb87473 [id=3d82ad2f9e184043aed4ec0b05df724f]). Expected version 1 got 0
I20251024 19:48:17.912062 1187 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 2d01572e51694d9bb58c3a59d4205102 (table default.Table_bf15d3c700774490b1860d3d8865a95e [id=2ea13e78137d47429d7cbe374d2113ef]). Expected version 1 got 0
I20251024 19:48:17.916723 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2602335327fc489fa154056c5ef5138f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:17.916947 1507 raft_consensus.cc:2468] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
W20251024 19:48:17.918356 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet cf18f375266b4448b3f67afd787df6ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf18f375266b4448b3f67afd787df6ad already in progress: creating tablet
W20251024 19:48:17.919533 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 839cef50795840fab564b94d88565f8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 839cef50795840fab564b94d88565f8c already in progress: creating tablet
W20251024 19:48:17.920508 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet fd52295b14b14e70bbdcb45f1bd849e4 because the tablet was not found. No further retry: Not found: Tablet not found: fd52295b14b14e70bbdcb45f1bd849e4
I20251024 19:48:17.921614 1775 tablet_replica.cc:333] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:17.922839 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a0027505cee405cb3a6feb17e08068e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.922945 1775 raft_consensus.cc:2243] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:17.923331 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19d83464fbf44f78881c7cae43a72091. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.923528 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef837960a0544202834a0ca509089b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.923719 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d8b8a1cff3540f7b7f13712425d534e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.926916 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16bc737104ab46b3b071a0760dbc847d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:17.927347 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332d8bd38f0b478d9592c781dfd5ca72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.923444 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16bc737104ab46b3b071a0760dbc847d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:17.928841 1505 raft_consensus.cc:3060] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.931540 1505 raft_consensus.cc:2468] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:17.932265 1775 raft_consensus.cc:2272] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:17.934859 1775 ts_tablet_manager.cc:1916] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:17.935426 1586 leader_election.cc:304] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:17.936842 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44305f9f044b4b14bbf66bbe612d14a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.943679 1767 tablet_bootstrap.cc:492] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:17.943900 1767 ts_tablet_manager.cc:1403] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.274s user 0.002s sys 0.008s
I20251024 19:48:17.944191 2923 raft_consensus.cc:2804] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:17.944351 2923 raft_consensus.cc:493] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:17.944434 2923 raft_consensus.cc:3060] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:17.944943 1767 raft_consensus.cc:359] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.945166 1767 raft_consensus.cc:385] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.945251 1767 raft_consensus.cc:740] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.945538 1767 consensus_queue.cc:260] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.946494 1767 ts_tablet_manager.cc:1434] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:17.947531 1767 tablet_bootstrap.cc:492] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:17.949021 1767 tablet_bootstrap.cc:654] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.957262 1768 tablet_bootstrap.cc:654] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.962055 2858 raft_consensus.cc:493] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.962266 2858 raft_consensus.cc:515] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.963455 2858 leader_election.cc:290] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.963626 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6424b2347ef642feb38a9596f2d2be1d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:17.964185 1447 leader_election.cc:343] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.963389 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6424b2347ef642feb38a9596f2d2be1d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:17.977365 1444 leader_election.cc:343] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:17.977488 1444 leader_election.cc:304] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:17.977880 2923 raft_consensus.cc:515] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.978127 2858 raft_consensus.cc:2749] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:17.978833 2923 leader_election.cc:290] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:17.979439 1768 tablet_bootstrap.cc:492] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:17.979579 1768 ts_tablet_manager.cc:1403] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.102s user 0.005s sys 0.000s
I20251024 19:48:17.980213 1768 raft_consensus.cc:359] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.980396 1768 raft_consensus.cc:385] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:17.980476 1768 raft_consensus.cc:740] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:17.980599 1774 ts_tablet_manager.cc:1929] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:17.980769 1774 log.cc:1199] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/fdfa5eea3b674a3389a156d1c72204df
W20251024 19:48:17.980789 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 45400us
I20251024 19:48:17.981226 1774 ts_tablet_manager.cc:1950] T fdfa5eea3b674a3389a156d1c72204df P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:17.980674 1768 consensus_queue.cc:260] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:17.981734 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 723) took 4863 ms. Trace:
I20251024 19:48:17.981987 1768 ts_tablet_manager.cc:1434] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:17.982242 1768 tablet_bootstrap.cc:492] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:17.982430 1766 tablet_bootstrap.cc:492] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:17.983487 3080 raft_consensus.cc:493] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:17.983631 1768 tablet_bootstrap.cc:654] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:17.983608 3080 raft_consensus.cc:515] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:17.984248 3080 leader_election.cc:290] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:17.984560 1586 leader_election.cc:304] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:17.969502 3097 log.cc:927] Time spent T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20251024 19:48:17.974222 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 3a6ef0c322e34c3590c522e018a710b6 (DEFAULT_TABLE table=default.Table_3f24dc3f9f2e410b992b0f2a2d022513 [id=f074f30826e24928819192eff4b33349]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.986999 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a6ef0c322e34c3590c522e018a710b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.988996 1776 ts_tablet_manager.cc:1929] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:17.989100 1776 log.cc:1199] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/b1fa439afe5a48a79557019687319af6
I20251024 19:48:17.989426 1776 ts_tablet_manager.cc:1950] T b1fa439afe5a48a79557019687319af6 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:17.990015 1766 ts_tablet_manager.cc:1403] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.102s user 0.001s sys 0.004s
I20251024 19:48:17.975251 1618 tablet_service.cc:1505] Processing CreateTablet for tablet fc50e9fbbefb43fda123ace9d1f32cf8 (DEFAULT_TABLE table=default.Table_34f366e7d1404f2e86acf32e02ae92ab [id=a00aaf99443a4765be2a095c601bc361]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:17.990895 3080 raft_consensus.cc:2804] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:17.990942 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc50e9fbbefb43fda123ace9d1f32cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:17.991003 3080 raft_consensus.cc:697] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:17.991250 3080 consensus_queue.cc:237] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:17.995978 1586 leader_election.cc:343] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:17.996500 1586 leader_election.cc:343] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:17.998844 1586 consensus_peers.cc:597] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:18.002118 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2602335327fc489fa154056c5ef5138f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:18.002321 1505 raft_consensus.cc:3060] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:18.005278 1447 consensus_peers.cc:597] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:18.005510 1768 tablet_bootstrap.cc:492] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:18.005685 1768 ts_tablet_manager.cc:1403] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20251024 19:48:18.006263 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2602335327fc489fa154056c5ef5138f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:18.006373 1768 raft_consensus.cc:359] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.006527 1768 raft_consensus.cc:385] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.006573 1768 raft_consensus.cc:740] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.006752 1768 consensus_queue.cc:260] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.007501 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6de2c21796f4bc4904f88b654c3baec" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:18.005272 1505 raft_consensus.cc:2468] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:17.976053 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 3ef6d244c498452ca79cd6c6d5c003df (DEFAULT_TABLE table=default.Table_a99c55b564b946b6a76c2b981640cdcd [id=a9095592179b462396c9963617da5e7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.008826 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef6d244c498452ca79cd6c6d5c003df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.008947 1768 ts_tablet_manager.cc:1434] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251024 19:48:18.009150 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6de2c21796f4bc4904f88b654c3baec" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:18.009241 1768 tablet_bootstrap.cc:492] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:18.009411 1505 raft_consensus.cc:2468] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:18.010845 1586 leader_election.cc:304] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:17.981846 1774 rpcz_store.cc:276] 1024 19:48:13.118774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.119307 (+ 533us) service_pool.cc:225] Handling call
1024 19:48:17.608650 (+4489343us) ts_tablet_manager.cc:1151] Deleting tablet fdfa5eea3b674a3389a156d1c72204df
1024 19:48:17.608661 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.940410 (+331749us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.980495 (+ 40085us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.981713 (+ 1218us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54596,"tablet-delete.queue_time_us":4434803}
I20251024 19:48:18.014776 1768 tablet_bootstrap.cc:654] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.017493 1418 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:18.014894 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 726) took 4887 ms. Trace:
I20251024 19:48:18.017848 1774 rpcz_store.cc:276] 1024 19:48:13.127510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.127801 (+ 291us) service_pool.cc:225] Handling call
1024 19:48:18.014767 (+4886966us) ts_tablet_manager.cc:1151] Deleting tablet d731403fbe0b4f92ac5899e13ea7bb19
1024 19:48:18.014776 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.014879 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4840800}
I20251024 19:48:18.018492 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 727) took 4890 ms. Trace:
I20251024 19:48:18.019833 2973 consensus_queue.cc:1048] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:18.020116 1774 rpcz_store.cc:276] 1024 19:48:13.128509 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.129037 (+ 528us) service_pool.cc:225] Handling call
1024 19:48:18.018362 (+4889325us) ts_tablet_manager.cc:1151] Deleting tablet 769f576810ab49368eb0fc7837b7da58
1024 19:48:18.018372 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.018476 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4844337}
I20251024 19:48:18.020673 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 728) took 4891 ms. Trace:
I20251024 19:48:18.020803 1774 rpcz_store.cc:276] 1024 19:48:13.129709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.131782 (+ 2073us) service_pool.cc:225] Handling call
1024 19:48:18.020572 (+4888790us) ts_tablet_manager.cc:1151] Deleting tablet fce7de9523fb4878a58aa8aef0c6b182
1024 19:48:18.020582 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.020659 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4846214}
I20251024 19:48:18.021248 1774 tablet_replica.cc:333] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:18.021540 1774 raft_consensus.cc:2243] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.021757 1774 raft_consensus.cc:2272] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.022392 1774 ts_tablet_manager.cc:1916] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.025972 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 187bc968c2b049028dd25dfb0264c0aa (table default.Table_31fb0d4536b94892aeceb91307c74837 [id=2bd12943c3e24d70a4f02626d7d58f6c]) successfully deleted
W20251024 19:48:18.026777 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b3e9960b7821421aa23e7ef5389f894a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3e9960b7821421aa23e7ef5389f894a already in progress: creating tablet
I20251024 19:48:18.017808 1618 tablet_service.cc:1505] Processing CreateTablet for tablet e879d4c72cda4bc781a2ebdf8cc3a179 (DEFAULT_TABLE table=default.Table_5f895593b7d0404c909c32f99e24d5b3 [id=186dcdefc76a4e62b86cd9b9e12a17f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.032642 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e879d4c72cda4bc781a2ebdf8cc3a179. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.032928 2920 raft_consensus.cc:2804] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:18.033254 2920 raft_consensus.cc:697] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:18.033547 2920 consensus_queue.cc:237] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:18.040702 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet e51e597b8fd044e393a3af07da25586c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e51e597b8fd044e393a3af07da25586c already in progress: creating tablet
W20251024 19:48:18.044061 1447 consensus_peers.cc:597] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:18.044745 1774 ts_tablet_manager.cc:1929] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:18.046737 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f89a3bbeb9b4d57a49078729110d951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.046967 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet 844638a6b6944d0394df3e78a8ea51d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.047070 1586 leader_election.cc:304] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:18.047312 3080 raft_consensus.cc:2804] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:18.047416 3080 raft_consensus.cc:493] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:18.047528 1774 log.cc:1199] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/d9bf08ccf7054fedb9efcbb00354c71a
I20251024 19:48:18.047536 3080 raft_consensus.cc:3060] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.048090 1774 ts_tablet_manager.cc:1950] T d9bf08ccf7054fedb9efcbb00354c71a P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:48:18.041290 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9684072368f14cc2942b4e240f90b0f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9684072368f14cc2942b4e240f90b0f2 already in progress: creating tablet
I20251024 19:48:18.048893 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 731) took 4916 ms. Trace:
I20251024 19:48:18.048978 1774 rpcz_store.cc:276] 1024 19:48:13.133331 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.133495 (+ 164us) service_pool.cc:225] Handling call
1024 19:48:18.021203 (+4887708us) ts_tablet_manager.cc:1151] Deleting tablet d9bf08ccf7054fedb9efcbb00354c71a
1024 19:48:18.021212 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.033104 (+ 11892us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.044620 (+ 11516us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.048877 (+ 4257us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49,"tablet-delete.queue_time_us":4846612}
I20251024 19:48:18.051110 3080 raft_consensus.cc:515] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:18.051826 3080 leader_election.cc:290] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:18.052575 1766 raft_consensus.cc:359] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:18.052791 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7cab9c17496947efbbe3cd9e5d8a49a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7cab9c17496947efbbe3cd9e5d8a49a8 already in progress: creating tablet
I20251024 19:48:18.052935 1766 raft_consensus.cc:385] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.053210 1766 raft_consensus.cc:740] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
W20251024 19:48:18.053238 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 292c31e7d4ce411bb336a1405ca43ba5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 292c31e7d4ce411bb336a1405ca43ba5 already in progress: creating tablet
I20251024 19:48:18.053496 1766 consensus_queue.cc:260] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:18.053839 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 734) took 4871 ms. Trace:
I20251024 19:48:18.053927 1774 rpcz_store.cc:276] 1024 19:48:13.182312 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.182348 (+ 36us) service_pool.cc:225] Handling call
1024 19:48:18.053702 (+4871354us) ts_tablet_manager.cc:1151] Deleting tablet b22b58404b8b441ba97ef536885a66e0
1024 19:48:18.053714 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.053820 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4804665}
I20251024 19:48:18.054287 1766 ts_tablet_manager.cc:1434] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.064s user 0.002s sys 0.000s
W20251024 19:48:18.054354 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 520060a9adca4bb0b8616bcc006dce52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 520060a9adca4bb0b8616bcc006dce52 already in progress: creating tablet
I20251024 19:48:18.054843 1774 tablet_replica.cc:333] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:18.055569 1774 raft_consensus.cc:2243] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.055899 1774 raft_consensus.cc:2272] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.055923 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 03d8ab62b47f474198841ea14a3b7b00 (DEFAULT_TABLE table=default.Table_1312dfdab750454a8f412ce6a66284ac [id=28dfd9c9c5cf470abbcfad936688bf94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.056427 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 03d8ab62b47f474198841ea14a3b7b00 (DEFAULT_TABLE table=default.Table_1312dfdab750454a8f412ce6a66284ac [id=28dfd9c9c5cf470abbcfad936688bf94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.056681 1774 ts_tablet_manager.cc:1916] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.056912 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03d8ab62b47f474198841ea14a3b7b00. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:18.057288 1586 leader_election.cc:343] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:18.057708 1586 leader_election.cc:343] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.058488 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6de2c21796f4bc4904f88b654c3baec" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:18.058844 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6de2c21796f4bc4904f88b654c3baec" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:18.059055 1505 raft_consensus.cc:3060] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:18.062549 1305 consensus_peers.cc:597] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2346c03802a245a2af8c086b626d660d. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:18.063449 1505 raft_consensus.cc:2468] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:18.028944 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 898) took 1047 ms. Trace:
I20251024 19:48:18.063905 1776 rpcz_store.cc:276] 1024 19:48:16.981940 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.985150 (+ 3210us) service_pool.cc:225] Handling call
1024 19:48:17.838357 (+853207us) ts_tablet_manager.cc:1151] Deleting tablet b1fa439afe5a48a79557019687319af6
1024 19:48:17.838370 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:17.871007 (+ 32637us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:17.988898 (+117891us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.028892 (+ 39994us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38357,"tablet-delete.queue_time_us":852995}
I20251024 19:48:18.064445 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet b1fa439afe5a48a79557019687319af6 (table default.Table_231e4d38da7a4fe4b4e6b33e854aa1d5 [id=77778be7b00c4eccbce4716fc4a864f7]) successfully deleted
I20251024 19:48:18.065515 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 904) took 1059 ms. Trace:
I20251024 19:48:18.065904 1776 rpcz_store.cc:276] 1024 19:48:17.005964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.006114 (+ 150us) service_pool.cc:225] Handling call
1024 19:48:18.065050 (+1058936us) ts_tablet_manager.cc:1151] Deleting tablet 0143a393711446fdbe6c798efa0c8936
1024 19:48:18.065397 (+ 347us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.065499 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049958}
I20251024 19:48:18.066428 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 903) took 1060 ms. Trace:
I20251024 19:48:18.066733 1776 rpcz_store.cc:276] 1024 19:48:17.005770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.006310 (+ 540us) service_pool.cc:225] Handling call
1024 19:48:18.066317 (+1060007us) ts_tablet_manager.cc:1151] Deleting tablet 3b8e8dbca79f4d72a2de51f105b209e4
1024 19:48:18.066329 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.066412 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1051080}
W20251024 19:48:18.066921 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 0143a393711446fdbe6c798efa0c8936 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0143a393711446fdbe6c798efa0c8936 already in progress: creating tablet
W20251024 19:48:18.067247 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3b8e8dbca79f4d72a2de51f105b209e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b8e8dbca79f4d72a2de51f105b209e4 already in progress: creating tablet
I20251024 19:48:18.067415 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 906) took 1061 ms. Trace:
I20251024 19:48:18.067523 1776 rpcz_store.cc:276] 1024 19:48:17.006215 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.006763 (+ 548us) service_pool.cc:225] Handling call
1024 19:48:18.067241 (+1060478us) ts_tablet_manager.cc:1151] Deleting tablet 4f00719c20934655bc744446194f7401
1024 19:48:18.067252 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.067396 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045551}
W20251024 19:48:18.068626 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4f00719c20934655bc744446194f7401 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f00719c20934655bc744446194f7401 already in progress: creating tablet
I20251024 19:48:18.069012 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 907) took 1062 ms. Trace:
I20251024 19:48:18.069128 1776 rpcz_store.cc:276] 1024 19:48:17.006326 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.006903 (+ 577us) service_pool.cc:225] Handling call
1024 19:48:18.068915 (+1062012us) ts_tablet_manager.cc:1151] Deleting tablet 4d0e4ffcdf184c5bafd99bf721b35951
1024 19:48:18.068925 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.068996 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045274}
W20251024 19:48:18.069658 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4d0e4ffcdf184c5bafd99bf721b35951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d0e4ffcdf184c5bafd99bf721b35951 already in progress: creating tablet
I20251024 19:48:18.070183 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet aeeaf256b1404653b690c712d9ef7a6d with delete_type TABLET_DATA_DELETED (replaced by tablet 6a93c672295a40e5a79501c0ff3d1a3c) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.070259 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 187bc968c2b049028dd25dfb0264c0aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.070407 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet aeeaf256b1404653b690c712d9ef7a6d with delete_type TABLET_DATA_DELETED (replaced by tablet 6a93c672295a40e5a79501c0ff3d1a3c) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.070699 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 910) took 1038 ms. Trace:
I20251024 19:48:18.070823 1776 rpcz_store.cc:276] 1024 19:48:17.032252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.032340 (+ 88us) service_pool.cc:225] Handling call
1024 19:48:18.070585 (+1038245us) ts_tablet_manager.cc:1151] Deleting tablet ca6b8783039a4f5fbbda65c129805642
1024 19:48:18.070595 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.070670 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1037980}
I20251024 19:48:18.071612 1774 ts_tablet_manager.cc:1929] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:18.071658 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 9d61e01cb1c240a0b4edd77621a86d75 (DEFAULT_TABLE table=default.Table_6011e1b8d3634908b57b079b4dea405e [id=782290308c2048ca81343ec18770b3ff]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:18.072187 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet ca6b8783039a4f5fbbda65c129805642 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca6b8783039a4f5fbbda65c129805642 already in progress: creating tablet
I20251024 19:48:18.072969 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 346cb0bb8a0d43f7a15b788a303a6bcf (DEFAULT_TABLE table=default.Table_532ecd1242ea40d29a9d1bcf6fa8a488 [id=8a0022a43f6a4a46b3a0dfd1942b5e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.073285 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 50cea450eda9485daba40bf875a1b90c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.074641 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 3ef6d244c498452ca79cd6c6d5c003df (DEFAULT_TABLE table=default.Table_a99c55b564b946b6a76c2b981640cdcd [id=a9095592179b462396c9963617da5e7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.071661 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 9d61e01cb1c240a0b4edd77621a86d75 (DEFAULT_TABLE table=default.Table_6011e1b8d3634908b57b079b4dea405e [id=782290308c2048ca81343ec18770b3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.072072 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 3887fc21726846279fecb02672b9bc07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.072714 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 346cb0bb8a0d43f7a15b788a303a6bcf (DEFAULT_TABLE table=default.Table_532ecd1242ea40d29a9d1bcf6fa8a488 [id=8a0022a43f6a4a46b3a0dfd1942b5e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.073295 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 3ef6d244c498452ca79cd6c6d5c003df (DEFAULT_TABLE table=default.Table_a99c55b564b946b6a76c2b981640cdcd [id=a9095592179b462396c9963617da5e7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.075294 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d61e01cb1c240a0b4edd77621a86d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.075454 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 346cb0bb8a0d43f7a15b788a303a6bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.075611 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef6d244c498452ca79cd6c6d5c003df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.081403 1774 log.cc:1199] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/ad055dc3387b414bb55d09b9683ba17e
I20251024 19:48:18.081980 1774 ts_tablet_manager.cc:1950] T ad055dc3387b414bb55d09b9683ba17e P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:18.082741 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 733) took 4900 ms. Trace:
I20251024 19:48:18.083253 1774 rpcz_store.cc:276] 1024 19:48:13.182197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.182293 (+ 96us) service_pool.cc:225] Handling call
1024 19:48:18.054797 (+4872504us) ts_tablet_manager.cc:1151] Deleting tablet ad055dc3387b414bb55d09b9683ba17e
1024 19:48:18.054806 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.066227 (+ 11421us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.071516 (+ 5289us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.082726 (+ 11210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4805663}
I20251024 19:48:18.084127 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 735) took 4753 ms. Trace:
I20251024 19:48:18.084527 1774 rpcz_store.cc:276] 1024 19:48:13.330769 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.330834 (+ 65us) service_pool.cc:225] Handling call
1024 19:48:18.083996 (+4753162us) ts_tablet_manager.cc:1151] Deleting tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4
1024 19:48:18.084005 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.084112 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4752959}
I20251024 19:48:18.085186 1774 tablet_replica.cc:333] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:18.085633 1774 raft_consensus.cc:2243] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.086151 1774 raft_consensus.cc:2272] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.086900 1774 ts_tablet_manager.cc:1916] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.091604 1366 raft_consensus.cc:3060] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.091894 1586 leader_election.cc:304] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:18.092257 1767 tablet_bootstrap.cc:492] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:18.092373 2923 raft_consensus.cc:2804] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:18.092665 2923 raft_consensus.cc:697] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:18.092886 2923 consensus_queue.cc:237] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:18.092985 1586 leader_election.cc:343] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.098457 1480 tablet_service.cc:1505] Processing CreateTablet for tablet 3a6ef0c322e34c3590c522e018a710b6 (DEFAULT_TABLE table=default.Table_3f24dc3f9f2e410b992b0f2a2d022513 [id=f074f30826e24928819192eff4b33349]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.098757 1488 tablet_service.cc:1505] Processing CreateTablet for tablet fc50e9fbbefb43fda123ace9d1f32cf8 (DEFAULT_TABLE table=default.Table_34f366e7d1404f2e86acf32e02ae92ab [id=a00aaf99443a4765be2a095c601bc361]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.098870 1339 tablet_service.cc:1505] Processing CreateTablet for tablet fc50e9fbbefb43fda123ace9d1f32cf8 (DEFAULT_TABLE table=default.Table_34f366e7d1404f2e86acf32e02ae92ab [id=a00aaf99443a4765be2a095c601bc361]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.099403 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc50e9fbbefb43fda123ace9d1f32cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.099555 1474 tablet_service.cc:1505] Processing CreateTablet for tablet e879d4c72cda4bc781a2ebdf8cc3a179 (DEFAULT_TABLE table=default.Table_5f895593b7d0404c909c32f99e24d5b3 [id=186dcdefc76a4e62b86cd9b9e12a17f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.100003 1340 tablet_service.cc:1505] Processing CreateTablet for tablet e879d4c72cda4bc781a2ebdf8cc3a179 (DEFAULT_TABLE table=default.Table_5f895593b7d0404c909c32f99e24d5b3 [id=186dcdefc76a4e62b86cd9b9e12a17f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.100374 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e879d4c72cda4bc781a2ebdf8cc3a179. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.100697 1768 tablet_bootstrap.cc:492] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:18.101148 1768 ts_tablet_manager.cc:1403] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.013s
I20251024 19:48:18.101683 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f630fd412ca4c4f8570359c1678e6af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:18.102056 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 88489.3us
I20251024 19:48:18.102810 1768 raft_consensus.cc:359] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.103623 1768 raft_consensus.cc:385] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.103849 1768 raft_consensus.cc:740] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.104223 1768 consensus_queue.cc:260] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.092370 1767 ts_tablet_manager.cc:1403] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.146s user 0.002s sys 0.002s
I20251024 19:48:18.105602 1768 ts_tablet_manager.cc:1434] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251024 19:48:18.106387 1767 raft_consensus.cc:359] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.106549 1767 raft_consensus.cc:385] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.106614 1767 raft_consensus.cc:740] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.106804 1767 consensus_queue.cc:260] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.107200 1768 tablet_bootstrap.cc:492] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:18.107259 1767 ts_tablet_manager.cc:1434] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:18.108673 1768 tablet_bootstrap.cc:654] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.109550 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c55267a23ded43cab7442fa75b611e12" } modify_external_catalogs: true
I20251024 19:48:18.109529 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9423eab92cc74554b6a926d3c438fd30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:18.109791 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c55267a23ded43cab7442fa75b611e12" } modify_external_catalogs: true
I20251024 19:48:18.109834 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d25c3d97a9a54883942fe92efe382f58" } modify_external_catalogs: true
W20251024 19:48:18.110180 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9423eab92cc74554b6a926d3c438fd30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.111483 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_669783bf95a24c658906c5ea84b932b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:18.112056 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_669783bf95a24c658906c5ea84b932b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:18.112142 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b8d30b2fdeec410fa582a05db0968ebb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8d30b2fdeec410fa582a05db0968ebb already in progress: creating tablet
I20251024 19:48:18.114212 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ada6a6efca404633ba28f55fdcdcebec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:18.114735 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ada6a6efca404633ba28f55fdcdcebec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.109879 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet be8126459ef541f39bc133f27c7d0f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.116688 1766 tablet_bootstrap.cc:492] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:18.116707 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_acc74be61adc4422bd3935ae750cf797" } new_table_name: "default.Table_89b86cd8af9f4deaac6fb553e6478e79" modify_external_catalogs: true
I20251024 19:48:18.118530 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_bb7717f8140d4bc6a90455b84c9906ea" } new_table_name: "default.Table_22c8767f9145492490b58591bd13abb4" modify_external_catalogs: true
I20251024 19:48:18.118705 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7fa0ffaa80294f7eb87d8b82770bb44c" } new_table_name: "default.Table_d7ded51d900c4f8cabddfdb0f3d069ec" modify_external_catalogs: true
I20251024 19:48:18.121459 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_23a248d91f5d45fbbf4bf104efd8bfb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:18.121762 1766 tablet_bootstrap.cc:654] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:18.122133 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23a248d91f5d45fbbf4bf104efd8bfb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.122607 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03d8ab62b47f474198841ea14a3b7b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.125388 1774 ts_tablet_manager.cc:1929] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:18.125497 1774 log.cc:1199] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/b79801594e95434cbb8fe7b75b14602e
I20251024 19:48:18.125896 1774 ts_tablet_manager.cc:1950] T b79801594e95434cbb8fe7b75b14602e P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:18.127813 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d61e01cb1c240a0b4edd77621a86d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.110039 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d25c3d97a9a54883942fe92efe382f58" } modify_external_catalogs: true
I20251024 19:48:18.131631 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 346cb0bb8a0d43f7a15b788a303a6bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.132161 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 3a6ef0c322e34c3590c522e018a710b6 (DEFAULT_TABLE table=default.Table_3f24dc3f9f2e410b992b0f2a2d022513 [id=f074f30826e24928819192eff4b33349]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.127508 3111 tablet.cc:1722] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:18.134119 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef6d244c498452ca79cd6c6d5c003df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.134441 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc50e9fbbefb43fda123ace9d1f32cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.135804 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a6ef0c322e34c3590c522e018a710b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.137741 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e879d4c72cda4bc781a2ebdf8cc3a179. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.138173 2973 raft_consensus.cc:493] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.138293 2973 raft_consensus.cc:515] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.138950 2973 leader_election.cc:290] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:18.139353 2973 raft_consensus.cc:493] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.139459 2973 raft_consensus.cc:515] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:18.139966 2973 leader_election.cc:290] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:18.144855 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet e7d6472ac7b046d89f56618fa4ed1811
I20251024 19:48:18.145999 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9e22d18b516543cf9c179d97a1665544"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:18.146777 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e22d18b516543cf9c179d97a1665544 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.147222 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6fd5c2ef64f24eb2a49b9e8f7715642b" } modify_external_catalogs: true
I20251024 19:48:18.147423 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6fd5c2ef64f24eb2a49b9e8f7715642b" } modify_external_catalogs: true
I20251024 19:48:18.150012 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0405765c6146cbbb187af96a60be2e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:18.150580 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 740) took 4802 ms. Trace:
I20251024 19:48:18.151364 1774 rpcz_store.cc:276] 1024 19:48:13.347711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.354863 (+ 7152us) service_pool.cc:225] Handling call
1024 19:48:18.085142 (+4730279us) ts_tablet_manager.cc:1151] Deleting tablet b79801594e95434cbb8fe7b75b14602e
1024 19:48:18.085150 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.113681 (+ 28531us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.125279 (+ 11598us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.150564 (+ 25285us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4651624}
I20251024 19:48:18.151638 2857 raft_consensus.cc:493] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.151794 2857 raft_consensus.cc:515] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.152051 1774 tablet_replica.cc:333] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:18.152662 2857 leader_election.cc:290] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:18.152310 1774 raft_consensus.cc:2243] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:18.153306 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet e7d6472ac7b046d89f56618fa4ed1811 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.153538 1774 raft_consensus.cc:2272] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.154384 1774 ts_tablet_manager.cc:1916] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.155766 2923 raft_consensus.cc:493] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.155944 2923 raft_consensus.cc:515] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.156845 2923 leader_election.cc:290] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:18.157249 1767 tablet_bootstrap.cc:492] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:48:18.164899 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b4f345b3b974681902f37b3975ea618 consensus state has no leader
I20251024 19:48:18.167793 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81a3de69a13c459b9b7c02761ab7261e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:18.172051 1444 leader_election.cc:343] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.174569 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a6ef0c322e34c3590c522e018a710b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.174626 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a1031d785b646b095ece9be666d2a77" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:18.175839 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81a3de69a13c459b9b7c02761ab7261e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:18.176834 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 51116.4us
W20251024 19:48:18.176915 1447 leader_election.cc:343] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:18.177217 1447 leader_election.cc:343] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.177314 1447 leader_election.cc:304] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:18.177415 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet fdfa5eea3b674a3389a156d1c72204df (table default.Table_73923b0648674b2891cbbf4b90d3f106 [id=96ea5a86f92b4d18b0d74f1092bd17b3]) successfully deleted
W20251024 19:48:18.177770 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet d731403fbe0b4f92ac5899e13ea7bb19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d731403fbe0b4f92ac5899e13ea7bb19 already in progress: creating tablet
W20251024 19:48:18.178076 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 769f576810ab49368eb0fc7837b7da58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 769f576810ab49368eb0fc7837b7da58 already in progress: creating tablet
I20251024 19:48:18.178275 2857 raft_consensus.cc:2749] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:18.178292 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet fce7de9523fb4878a58aa8aef0c6b182 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fce7de9523fb4878a58aa8aef0c6b182 already in progress: creating tablet
I20251024 19:48:18.178484 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet d9bf08ccf7054fedb9efcbb00354c71a (table default.Table_4a085504a0884b37b6787b3bcf7a847f [id=10833938cf88466b834698420d7c01e3]) successfully deleted
W20251024 19:48:18.178664 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b22b58404b8b441ba97ef536885a66e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b22b58404b8b441ba97ef536885a66e0 already in progress: creating tablet
I20251024 19:48:18.179203 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet ad055dc3387b414bb55d09b9683ba17e (table default.Table_3e341daa717746478259c784488beecf [id=c3de6937e93a48dcb88cabee59a09d32]) successfully deleted
W20251024 19:48:18.179454 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b52ed5ce68a4cb2ae0dfa4c104e99a4 already in progress: creating tablet
I20251024 19:48:18.179793 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a1031d785b646b095ece9be666d2a77" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:18.180017 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet b79801594e95434cbb8fe7b75b14602e (table default.Table_aa7c283553a841e9ab2fa21fb6485a11 [id=7ec9391c52b1424eb52850f2c6ba4779]) successfully deleted
W20251024 19:48:18.180585 1444 leader_election.cc:343] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.181075 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet e7d6472ac7b046d89f56618fa4ed1811 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.181934 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet e7d6472ac7b046d89f56618fa4ed1811 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.188859 1768 tablet_bootstrap.cc:492] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:18.189029 1768 ts_tablet_manager.cc:1403] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.082s user 0.006s sys 0.007s
W20251024 19:48:18.189484 1447 leader_election.cc:343] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.189640 1447 leader_election.cc:304] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:18.190126 2858 raft_consensus.cc:2749] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:18.194867 1768 raft_consensus.cc:359] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:18.195060 1768 raft_consensus.cc:385] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.195132 1768 raft_consensus.cc:740] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.195341 1768 consensus_queue.cc:260] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:18.195916 1768 ts_tablet_manager.cc:1434] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251024 19:48:18.197252 1768 tablet_bootstrap.cc:492] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:18.200785 1768 tablet_bootstrap.cc:654] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.201223 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0405765c6146cbbb187af96a60be2e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:18.201764 1444 leader_election.cc:343] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.201890 1444 leader_election.cc:304] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:18.202162 2858 raft_consensus.cc:2749] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:18.110128 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 16bc737104ab46b3b071a0760dbc847d with delete_type TABLET_DATA_DELETED (replaced by tablet 385e71e672154e399f14321eac0d8625) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.214818 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be8126459ef541f39bc133f27c7d0f59" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:18.215070 1366 raft_consensus.cc:2468] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:18.216063 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 918) took 1003 ms. Trace:
I20251024 19:48:18.216171 1776 rpcz_store.cc:276] 1024 19:48:17.212806 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.212864 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:18.112521 (+899657us) ts_tablet_manager.cc:1151] Deleting tablet eb6b7df6176e4bccacf7a9059cee5477
1024 19:48:18.215895 (+103374us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x61800000d518. stack: 00007fea3c4169f6 00007fea3c4167b0 00007fea3b3f23ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fea3c3ff2fd 00007fea3c40036e 00007fea50032c9d 00007fea4ffb1203 00007fea4c1e4646 00007fea4c1e59c9 00007fea4c2192d4 00007fea4c1f9d70 00007fea4c1f9afd 0000000000a8867c
1024 19:48:18.215908 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.216034 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":236695296,"tablet-delete.queue_time_us":899490}
I20251024 19:48:18.216652 1776 tablet_replica.cc:333] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:18.216849 1776 raft_consensus.cc:2243] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
W20251024 19:48:18.217058 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet eb6b7df6176e4bccacf7a9059cee5477 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb6b7df6176e4bccacf7a9059cee5477 already in progress: creating tablet
I20251024 19:48:18.217163 1776 raft_consensus.cc:2272] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:18.217463 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 40338a7939be4240b44dc54cc77ef020 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40338a7939be4240b44dc54cc77ef020 already in progress: creating tablet
I20251024 19:48:18.220996 1586 leader_election.cc:304] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:18.221347 2923 raft_consensus.cc:2804] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:18.221426 2923 raft_consensus.cc:493] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:18.221482 2923 raft_consensus.cc:3060] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.223449 1767 tablet_bootstrap.cc:654] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.223628 1766 tablet_bootstrap.cc:492] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:18.223804 1766 ts_tablet_manager.cc:1403] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.107s user 0.002s sys 0.002s
I20251024 19:48:18.224686 2923 raft_consensus.cc:515] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.224840 1766 raft_consensus.cc:359] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:18.224992 1766 raft_consensus.cc:385] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.225039 1766 raft_consensus.cc:740] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.225329 2923 leader_election.cc:290] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:18.225286 1766 consensus_queue.cc:260] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:18.229147 1775 ts_tablet_manager.cc:1929] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:18.229292 1775 log.cc:1199] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/b1fa439afe5a48a79557019687319af6
I20251024 19:48:18.229813 1766 ts_tablet_manager.cc:1434] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251024 19:48:18.230805 1768 tablet_bootstrap.cc:492] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:18.231009 1768 ts_tablet_manager.cc:1403] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.002s
W20251024 19:48:18.227998 1586 leader_election.cc:343] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: be8126459ef541f39bc133f27c7d0f59
I20251024 19:48:18.232004 1768 raft_consensus.cc:359] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.232278 1768 raft_consensus.cc:385] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.232384 1768 raft_consensus.cc:740] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.232633 1768 consensus_queue.cc:260] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.233258 1767 tablet_bootstrap.cc:492] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:18.233275 1768 ts_tablet_manager.cc:1434] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:18.233548 1768 tablet_bootstrap.cc:492] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:18.235289 1768 tablet_bootstrap.cc:654] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.236271 1766 tablet_bootstrap.cc:492] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:18.237968 1775 ts_tablet_manager.cc:1950] T b1fa439afe5a48a79557019687319af6 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:18.232301 1774 ts_tablet_manager.cc:1929] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251024 19:48:18.238221 1774 log.cc:1199] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/2346c03802a245a2af8c086b626d660d
I20251024 19:48:18.238651 1774 ts_tablet_manager.cc:1950] T 2346c03802a245a2af8c086b626d660d P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:18.239050 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 745) took 4580 ms. Trace:
I20251024 19:48:18.239183 1774 rpcz_store.cc:276] 1024 19:48:13.658468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.659155 (+ 687us) service_pool.cc:225] Handling call
1024 19:48:18.151989 (+4492834us) ts_tablet_manager.cc:1151] Deleting tablet 2346c03802a245a2af8c086b626d660d
1024 19:48:18.152005 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.187573 (+ 35568us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.232178 (+ 44605us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.239033 (+ 6855us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4491589}
I20251024 19:48:18.239684 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 746) took 4530 ms. Trace:
I20251024 19:48:18.239755 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 2346c03802a245a2af8c086b626d660d (table default.Table_1865562320ef412799f93a15c66ade0c [id=aadca5a1b6d4494aa37f8a8cc76ad12a]) successfully deleted
I20251024 19:48:18.239985 1774 rpcz_store.cc:276] 1024 19:48:13.709346 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.731045 (+ 21699us) service_pool.cc:225] Handling call
1024 19:48:18.239566 (+4508521us) ts_tablet_manager.cc:1151] Deleting tablet 73278db2097d4001a538d62dacb97c9b
1024 19:48:18.239577 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.239669 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4508272}
I20251024 19:48:18.239836 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be8126459ef541f39bc133f27c7d0f59" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:18.241765 1364 raft_consensus.cc:3060] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.241832 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 747) took 4532 ms. Trace:
W20251024 19:48:18.241927 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 73278db2097d4001a538d62dacb97c9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73278db2097d4001a538d62dacb97c9b already in progress: creating tablet
I20251024 19:48:18.241909 1774 rpcz_store.cc:276] 1024 19:48:13.709457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.724609 (+ 15152us) service_pool.cc:225] Handling call
1024 19:48:18.241712 (+4517103us) ts_tablet_manager.cc:1151] Deleting tablet 9377725db312439e8a42dbd46da84a9d
1024 19:48:18.241721 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.241816 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4492977}
I20251024 19:48:18.242220 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 748) took 4493 ms. Trace:
W20251024 19:48:18.242306 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9377725db312439e8a42dbd46da84a9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9377725db312439e8a42dbd46da84a9d already in progress: creating tablet
I20251024 19:48:18.242285 1774 rpcz_store.cc:276] 1024 19:48:13.748677 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.749937 (+ 1260us) service_pool.cc:225] Handling call
1024 19:48:18.242122 (+4492185us) ts_tablet_manager.cc:1151] Deleting tablet 2764689179504e079924fde7e425dbcd
1024 19:48:18.242132 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.242202 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4491496}
I20251024 19:48:18.242529 1774 tablet_replica.cc:333] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
W20251024 19:48:18.242692 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2764689179504e079924fde7e425dbcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2764689179504e079924fde7e425dbcd already in progress: creating tablet
I20251024 19:48:18.242719 1774 raft_consensus.cc:2243] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.242877 1774 raft_consensus.cc:2272] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.244714 1364 raft_consensus.cc:2468] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:18.245517 1774 ts_tablet_manager.cc:1916] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.257718 2858 raft_consensus.cc:493] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.258023 2858 raft_consensus.cc:515] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:18.258610 2858 leader_election.cc:290] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:18.238035 1766 tablet_bootstrap.cc:654] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.266036 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "385e71e672154e399f14321eac0d8625" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:18.266647 1447 leader_election.cc:343] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.267186 1767 ts_tablet_manager.cc:1403] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.000s
I20251024 19:48:18.267521 1768 tablet_bootstrap.cc:492] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:18.267647 1768 ts_tablet_manager.cc:1403] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.004s
I20251024 19:48:18.268142 1767 raft_consensus.cc:359] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.268388 1767 raft_consensus.cc:385] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.268343 1768 raft_consensus.cc:359] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.268510 1768 raft_consensus.cc:385] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.268566 1768 raft_consensus.cc:740] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.268478 1767 raft_consensus.cc:740] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.268779 1768 consensus_queue.cc:260] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.268906 1767 consensus_queue.cc:260] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.269675 1767 ts_tablet_manager.cc:1434] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:18.270063 1767 tablet_bootstrap.cc:492] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:48:18.270794 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4289d6b43f32464696118e5840bad849 consensus state has no leader
I20251024 19:48:18.272341 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 81e3e2f8082d4c229e617f34f5eb5a78
I20251024 19:48:18.272922 1768 ts_tablet_manager.cc:1434] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251024 19:48:18.273193 1768 tablet_bootstrap.cc:492] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:18.273362 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec0a8fc1b67b4314ad8d305a9d13a1b6 consensus state has no leader
I20251024 19:48:18.276145 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7a65fc59de5346d6a9344c5310455e53
I20251024 19:48:18.276656 2858 raft_consensus.cc:493] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.276647 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_19e0e2ffa4d5450f9e3287136c9f5b32" } modify_external_catalogs: true
I20251024 19:48:18.276688 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9db3b592a2a64917b67ad65189a48a88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:18.276803 2858 raft_consensus.cc:515] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.276863 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_19e0e2ffa4d5450f9e3287136c9f5b32" } modify_external_catalogs: true
I20251024 19:48:18.277022 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_33762464873048acb73f8d26e167301e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:18.277315 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9db3b592a2a64917b67ad65189a48a88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.277434 2858 leader_election.cc:290] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:18.279208 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_17f78ac6efb148f896b6fc2a2abad27a" } new_table_name: "default.Table_be71c3e6e12d4afe9bcd0e4afa60a249" modify_external_catalogs: true
W20251024 19:48:18.277530 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33762464873048acb73f8d26e167301e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.282297 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee68243b4a664575b679950603bb250e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:18.282358 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6cfafb2b827d4dba9de0de5bb5472304"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:18.282454 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4d0d05989b3844cd8813c527af4862a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:18.282819 1447 leader_election.cc:343] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:18.282925 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cfafb2b827d4dba9de0de5bb5472304 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.283098 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee68243b4a664575b679950603bb250e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:18.283336 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_12fe00934de64b41af4f5fa5980dd476"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:18.283655 1444 leader_election.cc:343] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:18.283743 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12fe00934de64b41af4f5fa5980dd476 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.283756 1444 leader_election.cc:304] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:18.284145 2858 raft_consensus.cc:2749] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:18.284128 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e739c763ff31475a9a277870eb5d0132" } modify_external_catalogs: true
I20251024 19:48:18.284286 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e739c763ff31475a9a277870eb5d0132" } modify_external_catalogs: true
W20251024 19:48:18.282933 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d0d05989b3844cd8813c527af4862a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.286264 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a65fc59de5346d6a9344c5310455e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.286537 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a65fc59de5346d6a9344c5310455e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.286988 2920 raft_consensus.cc:493] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.287123 2920 raft_consensus.cc:515] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.287673 2920 leader_election.cc:290] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:18.287909 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a65fc59de5346d6a9344c5310455e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.288175 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 81e3e2f8082d4c229e617f34f5eb5a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.289183 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 81e3e2f8082d4c229e617f34f5eb5a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.291977 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f16ef86c5eb24489a66d6cd4bfa804fa
I20251024 19:48:18.292069 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 81e3e2f8082d4c229e617f34f5eb5a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.292654 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet d2c92975cf66441f9ac90c19772d3db3 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:18.293169 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet f16ef86c5eb24489a66d6cd4bfa804fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.295061 2920 raft_consensus.cc:493] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.295202 2920 raft_consensus.cc:515] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.295835 2920 leader_election.cc:290] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:18.296867 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_35b42b8f59b042ba9161297fca0d9bd3" } new_table_name: "default.Table_2c976ad07a404975b8cd35a9843c8162" modify_external_catalogs: true
I20251024 19:48:18.297155 1766 tablet_bootstrap.cc:492] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:18.297346 1766 ts_tablet_manager.cc:1403] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.067s user 0.005s sys 0.000s
W20251024 19:48:18.298712 1305 consensus_peers.cc:597] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f51c56337ebb450aaf62dcc81ef63ae8. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:18.299707 1766 raft_consensus.cc:359] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:18.299983 1766 raft_consensus.cc:385] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.300078 1766 raft_consensus.cc:740] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.300328 1766 consensus_queue.cc:260] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:18.300400 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet f16ef86c5eb24489a66d6cd4bfa804fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.302116 1766 ts_tablet_manager.cc:1434] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251024 19:48:18.303469 1768 tablet_bootstrap.cc:654] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.303841 1189 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet f16aab16b6de4a0284ce2dfccfe910a4
I20251024 19:48:18.306622 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 835) took 3468 ms. Trace:
I20251024 19:48:18.306794 1775 rpcz_store.cc:276] 1024 19:48:14.837843 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.837919 (+ 76us) service_pool.cc:225] Handling call
1024 19:48:17.901640 (+3063721us) ts_tablet_manager.cc:1151] Deleting tablet b1fa439afe5a48a79557019687319af6
1024 19:48:17.901649 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.031621 (+129972us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.225887 (+194266us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.306603 (+ 80716us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3063474}
W20251024 19:48:18.307156 1308 consensus_peers.cc:597] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f51c56337ebb450aaf62dcc81ef63ae8. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:18.307343 1775 tablet_replica.cc:333] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:18.307422 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet b1fa439afe5a48a79557019687319af6 (table default.Table_231e4d38da7a4fe4b4e6b33e854aa1d5 [id=77778be7b00c4eccbce4716fc4a864f7]) successfully deleted
I20251024 19:48:18.307658 1775 raft_consensus.cc:2243] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.307715 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet f16ef86c5eb24489a66d6cd4bfa804fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.307945 1775 raft_consensus.cc:2272] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.308496 1775 ts_tablet_manager.cc:1916] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.309401 2858 consensus_queue.cc:1048] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:18.311498 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87575045817948d9b4980944b2e62614 consensus state has no leader
I20251024 19:48:18.239490 1776 ts_tablet_manager.cc:1916] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.247365 1586 leader_election.cc:304] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:18.261631 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "385e71e672154e399f14321eac0d8625" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:18.315789 1444 leader_election.cc:343] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.315920 1444 leader_election.cc:304] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:18.316362 2858 raft_consensus.cc:2749] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:18.316713 2920 raft_consensus.cc:2804] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:18.316847 2920 raft_consensus.cc:697] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:18.317111 2920 consensus_queue.cc:237] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:18.319741 1447 consensus_peers.cc:597] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:18.330873 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 83538.4us
W20251024 19:48:18.331220 1586 leader_election.cc:343] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: be8126459ef541f39bc133f27c7d0f59
I20251024 19:48:18.333505 2926 raft_consensus.cc:493] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.333676 2926 raft_consensus.cc:515] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.334290 2926 leader_election.cc:290] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:18.335793 1768 tablet_bootstrap.cc:492] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:18.335939 1768 ts_tablet_manager.cc:1403] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20251024 19:48:18.335917 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94fc8c5716af47faa4e831f05fce8610" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:18.336145 1505 raft_consensus.cc:2468] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:18.336683 1768 raft_consensus.cc:359] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.336845 1768 raft_consensus.cc:385] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.336894 1768 raft_consensus.cc:740] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.337105 1768 consensus_queue.cc:260] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.339866 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94fc8c5716af47faa4e831f05fce8610" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:18.340790 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62667092d7e244cca1020ec220ea9124" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:18.340976 1505 raft_consensus.cc:2468] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:18.342036 1768 ts_tablet_manager.cc:1434] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251024 19:48:18.342038 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62667092d7e244cca1020ec220ea9124" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:18.342839 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f630fd412ca4c4f8570359c1678e6af" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:18.343001 1505 raft_consensus.cc:2468] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:18.343639 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f630fd412ca4c4f8570359c1678e6af" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:18.345947 1768 tablet_bootstrap.cc:492] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:18.350862 1774 ts_tablet_manager.cc:1929] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:18.351061 1774 log.cc:1199] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/758df87acbdb4adaa83e1ecbc44e7cf1
I20251024 19:48:18.351504 1774 ts_tablet_manager.cc:1950] T 758df87acbdb4adaa83e1ecbc44e7cf1 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:18.351712 1767 tablet_bootstrap.cc:654] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.352913 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet 36ccb0cfba7a4a14b36d38897f4a9ef0 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:18.353613 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 749) took 4486 ms. Trace:
I20251024 19:48:18.354236 1774 rpcz_store.cc:276] 1024 19:48:13.867166 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:13.867253 (+ 87us) service_pool.cc:225] Handling call
1024 19:48:18.242489 (+4375236us) ts_tablet_manager.cc:1151] Deleting tablet 758df87acbdb4adaa83e1ecbc44e7cf1
1024 19:48:18.242499 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.329733 (+ 87234us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.350765 (+ 21032us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.353557 (+ 2792us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4374956}
I20251024 19:48:18.354887 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 758df87acbdb4adaa83e1ecbc44e7cf1 (table default.Table_003cb0c156394f9b8b0ba1b4ac52ce4c [id=b81a7d71911844c1be495ade2ceaf1d4]) successfully deleted
I20251024 19:48:18.354084 1768 tablet_bootstrap.cc:654] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:18.355166 1586 leader_election.cc:343] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:18.355443 1586 leader_election.cc:343] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:18.355679 1586 leader_election.cc:343] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.356211 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 756) took 4035 ms. Trace:
I20251024 19:48:18.356333 1774 rpcz_store.cc:276] 1024 19:48:14.320405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.320467 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:18.356085 (+4035618us) ts_tablet_manager.cc:1151] Deleting tablet b19cce078dc7426aacbb60f45379e506
1024 19:48:18.356097 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.356194 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4035372}
W20251024 19:48:18.356910 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b19cce078dc7426aacbb60f45379e506 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b19cce078dc7426aacbb60f45379e506 already in progress: creating tablet
I20251024 19:48:18.357606 1774 tablet_replica.cc:333] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:18.358461 1774 raft_consensus.cc:2243] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.358719 3111 tablet.cc:1722] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:18.359169 1586 leader_election.cc:304] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:18.359968 2926 raft_consensus.cc:2804] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:18.360045 2926 raft_consensus.cc:493] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:18.360106 2926 raft_consensus.cc:3060] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.362855 2926 raft_consensus.cc:515] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.363442 2926 leader_election.cc:290] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:18.364468 1586 leader_election.cc:304] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:18.358703 1774 raft_consensus.cc:2272] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.374816 2973 consensus_queue.cc:1048] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:18.375268 1774 ts_tablet_manager.cc:1916] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:18.379570 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc35c0a2b36c45ffa2fe4ad00abccfd1 consensus state has no leader
I20251024 19:48:18.380363 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 81a3de69a13c459b9b7c02761ab7261e (table default.Table_b5a66c4e9cd8408782690b66cdb87473 [id=3d82ad2f9e184043aed4ec0b05df724f]). Expected version 1 got 0
I20251024 19:48:18.380343 1766 tablet_bootstrap.cc:492] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:18.382092 1192 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 067631b128fd4a1e9ab72c9507dab6c4 (table default.Table_814a6dad46d54945b70e652b7e0f0032 [id=3f394c59d82b44928beb22f42312a4a3]). Expected version 1 got 0
I20251024 19:48:18.382344 1192 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet e03c488d0f264684b0897a81a04d7466 (table default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8]). Expected version 2 got 0
I20251024 19:48:18.382648 1192 catalog_manager.cc:5649] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } }
I20251024 19:48:18.383140 1192 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet d6de2c21796f4bc4904f88b654c3baec (table default.Table_1a079b981b694617b3baf9897dfb37ae [id=18ec057e920e4c21ae8cbe298058b6f3]). Expected version 2 got 0
I20251024 19:48:18.383458 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_266203ab5d8d40979ab181d5715a4348"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:18.383879 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 977b4070419948d4910e2b69e01ff132 (DEFAULT_TABLE table=default.Table_fe56cf0299064493aa6b4f993e0ad929 [id=a860277e104846ac8cec903d14a7020c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:18.384225 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_266203ab5d8d40979ab181d5715a4348 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.384372 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 977b4070419948d4910e2b69e01ff132 (DEFAULT_TABLE table=default.Table_fe56cf0299064493aa6b4f993e0ad929 [id=a860277e104846ac8cec903d14a7020c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.384794 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 977b4070419948d4910e2b69e01ff132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.386246 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 977b4070419948d4910e2b69e01ff132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.386317 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_27407bc1a5e3473abd1a59db49046843" } modify_external_catalogs: true
I20251024 19:48:18.386725 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_27407bc1a5e3473abd1a59db49046843" } modify_external_catalogs: true
I20251024 19:48:18.388252 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_83b888420618403f86e347bdb1c9f038"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:18.388480 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_bf5080a43b0841818ccdbb98b65e05ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:18.388752 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83b888420618403f86e347bdb1c9f038 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:18.388996 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf5080a43b0841818ccdbb98b65e05ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.389218 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet f83b9290f00145a780318a43249eec12 with delete_type TABLET_DATA_DELETED (replaced by tablet 41176885447e42409ee4363bb4d903f2) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.389937 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 4193750331874d7ba9bbe6279c3a8e37 with delete_type TABLET_DATA_DELETED (replaced by tablet ec2ffe32f2914ef59d5d77056fbb7d16) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.390100 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 2602335327fc489fa154056c5ef5138f with delete_type TABLET_DATA_DELETED (replaced by tablet 74029a7d7cdc4503a833c1758887b684) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.392197 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 2869d5a7a29d479d891c3a4ba50e1ff2 (DEFAULT_TABLE table=default.Table_468b839284304d4eb9920feeeb49bc18 [id=8e3f7bc3fff24aa1a663cbc6cdb8d08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.392539 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2869d5a7a29d479d891c3a4ba50e1ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.398717 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 36ccb0cfba7a4a14b36d38897f4a9ef0
I20251024 19:48:18.398911 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e6db58fac5e4af9b27bde6c223aa845 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.399267 1190 catalog_manager.cc:6719] ReplaceTablet: tablet 36ccb0cfba7a4a14b36d38897f4a9ef0 deleted and replaced by tablet 9a94c856db4141cfb7ada7515d5b6e9e
I20251024 19:48:18.400327 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 2869d5a7a29d479d891c3a4ba50e1ff2 (DEFAULT_TABLE table=default.Table_468b839284304d4eb9920feeeb49bc18 [id=8e3f7bc3fff24aa1a663cbc6cdb8d08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.400728 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2869d5a7a29d479d891c3a4ba50e1ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.403267 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b95989b98c77472faa7bf940cfa60432
I20251024 19:48:18.411795 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 36ccb0cfba7a4a14b36d38897f4a9ef0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a94c856db4141cfb7ada7515d5b6e9e) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.412804 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet b95989b98c77472faa7bf940cfa60432 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.412989 1335 tablet_service.cc:1505] Processing CreateTablet for tablet c8a2b5a84ce24935a06c8e8d7a70cb59 (DEFAULT_TABLE table=default.Table_dbb22386d4e04757bc3b5e73e46c6e1a [id=6eb631a1bb944115b56c51257a0bb716]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.413367 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8a2b5a84ce24935a06c8e8d7a70cb59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.414157 1339 tablet_service.cc:1505] Processing CreateTablet for tablet f2fe735d85414bcda79ea4399adc041f (DEFAULT_TABLE table=default.Table_d2f10524d22141078abbeeea11a129ef [id=5354680146fb4514a83484157f46a1fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.414454 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2fe735d85414bcda79ea4399adc041f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.414746 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 36ccb0cfba7a4a14b36d38897f4a9ef0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a94c856db4141cfb7ada7515d5b6e9e) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.415094 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet ee68243b4a664575b679950603bb250e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.415408 1483 tablet_service.cc:1505] Processing CreateTablet for tablet c8a2b5a84ce24935a06c8e8d7a70cb59 (DEFAULT_TABLE table=default.Table_dbb22386d4e04757bc3b5e73e46c6e1a [id=6eb631a1bb944115b56c51257a0bb716]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.415726 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8a2b5a84ce24935a06c8e8d7a70cb59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.415927 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 366c17dbe45a4f8f9830dca68f621121 (DEFAULT_TABLE table=default.Table_6233280a74da4d1981019506fd1be2f3 [id=c6fd443442b2458ea8842a1ec8b67081]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.416249 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 366c17dbe45a4f8f9830dca68f621121. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.417042 1480 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf4596b333244cc9e9e35a128721e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.417884 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 6d1172e69d78456dbdbf0dba34d80db7 (DEFAULT_TABLE table=default.Table_303e87d506b64ecdbb134e0476487dec [id=aaaec8f080314033858bc8320c10cab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.418200 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d1172e69d78456dbdbf0dba34d80db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.418876 1489 tablet_service.cc:1505] Processing CreateTablet for tablet f2fe735d85414bcda79ea4399adc041f (DEFAULT_TABLE table=default.Table_d2f10524d22141078abbeeea11a129ef [id=5354680146fb4514a83484157f46a1fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.419188 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2fe735d85414bcda79ea4399adc041f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.421674 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet b95989b98c77472faa7bf940cfa60432 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.421813 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 8ecb196eba1e4317af472ef753894bb9 (DEFAULT_TABLE table=default.Table_88a506f2737145189da07becb2912c29 [id=0fda2b1307fe40d1923326929f2bee2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.421864 1474 tablet_service.cc:1505] Processing CreateTablet for tablet f716f8585d4447629262089e6d7cefc9 (DEFAULT_TABLE table=default.Table_77d2dbff62c647e2b909bc059207d895 [id=1fc5cb1865c6446ebef50b6b74ab6359]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.421980 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 995abf5a55b7445999c2b2c161610019 (DEFAULT_TABLE table=default.Table_658377c2e7954211a0e56c012dc08bf6 [id=0a1b6e5e13b340a69060753aa21590c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.422112 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ecb196eba1e4317af472ef753894bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.422245 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 995abf5a55b7445999c2b2c161610019. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.425712 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f716f8585d4447629262089e6d7cefc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.425853 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 2fb5f8d81a924240a53715f6a43ab87f (DEFAULT_TABLE table=default.Table_2acb8ac85b4e4eae9ec2e97990416a3f [id=15352b76444849e2a89000475e102023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.426028 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 4ec69ffe2c024d1a8596a93183d1458b (DEFAULT_TABLE table=default.Table_9ff460fc04ce46c0a4ae4163efee611b [id=c748eed9ebab4b7c9f43f61c12419808]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.426162 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb5f8d81a924240a53715f6a43ab87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.426288 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec69ffe2c024d1a8596a93183d1458b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.427006 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 96e8588e092f45c2b10f4e1e97408c47 (DEFAULT_TABLE table=default.Table_6f21add4230f4d1692ce71616d05ff7a [id=55f8fb6f42cb41ca8aa74748a992413d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.427320 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e8588e092f45c2b10f4e1e97408c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.429874 1472 tablet_service.cc:1505] Processing CreateTablet for tablet a9b1f6e4597045bfbe4a4645f105d8c8 (DEFAULT_TABLE table=default.Table_074163dbc16146cf81fc6db79bd636b0 [id=7f1c07c184f84adcba5735f76bdf98a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.429880 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 4ec69ffe2c024d1a8596a93183d1458b (DEFAULT_TABLE table=default.Table_9ff460fc04ce46c0a4ae4163efee611b [id=c748eed9ebab4b7c9f43f61c12419808]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.430078 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 8ecb196eba1e4317af472ef753894bb9 (DEFAULT_TABLE table=default.Table_88a506f2737145189da07becb2912c29 [id=0fda2b1307fe40d1923326929f2bee2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.430119 1333 tablet_service.cc:1505] Processing CreateTablet for tablet a9b1f6e4597045bfbe4a4645f105d8c8 (DEFAULT_TABLE table=default.Table_074163dbc16146cf81fc6db79bd636b0 [id=7f1c07c184f84adcba5735f76bdf98a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.430231 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec69ffe2c024d1a8596a93183d1458b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.430231 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9b1f6e4597045bfbe4a4645f105d8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.430444 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ecb196eba1e4317af472ef753894bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.430456 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 995abf5a55b7445999c2b2c161610019 (DEFAULT_TABLE table=default.Table_658377c2e7954211a0e56c012dc08bf6 [id=0a1b6e5e13b340a69060753aa21590c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.425853 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 96e8588e092f45c2b10f4e1e97408c47 (DEFAULT_TABLE table=default.Table_6f21add4230f4d1692ce71616d05ff7a [id=55f8fb6f42cb41ca8aa74748a992413d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.430889 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 995abf5a55b7445999c2b2c161610019. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.430953 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e8588e092f45c2b10f4e1e97408c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.433806 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 2fb5f8d81a924240a53715f6a43ab87f (DEFAULT_TABLE table=default.Table_2acb8ac85b4e4eae9ec2e97990416a3f [id=15352b76444849e2a89000475e102023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.429880 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 366c17dbe45a4f8f9830dca68f621121 (DEFAULT_TABLE table=default.Table_6233280a74da4d1981019506fd1be2f3 [id=c6fd443442b2458ea8842a1ec8b67081]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.434765 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a1031d785b646b095ece9be666d2a77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.436825 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb5f8d81a924240a53715f6a43ab87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.437013 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9b1f6e4597045bfbe4a4645f105d8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.441712 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 366c17dbe45a4f8f9830dca68f621121. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.442183 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 6424b2347ef642feb38a9596f2d2be1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.447274 1345 tablet_service.cc:1505] Processing CreateTablet for tablet f716f8585d4447629262089e6d7cefc9 (DEFAULT_TABLE table=default.Table_77d2dbff62c647e2b909bc059207d895 [id=1fc5cb1865c6446ebef50b6b74ab6359]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.447578 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 6d1172e69d78456dbdbf0dba34d80db7 (DEFAULT_TABLE table=default.Table_303e87d506b64ecdbb134e0476487dec [id=aaaec8f080314033858bc8320c10cab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.447664 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f716f8585d4447629262089e6d7cefc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.447878 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d1172e69d78456dbdbf0dba34d80db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.452032 2936 raft_consensus.cc:493] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af3952de050140d383f6b0de7666fa97)
I20251024 19:48:18.452194 2936 raft_consensus.cc:515] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.452863 2936 leader_election.cc:290] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:18.453552 1308 leader_election.cc:343] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 27b32126d04e480f8db7b2b20aba5873
W20251024 19:48:18.456640 1305 leader_election.cc:343] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 27b32126d04e480f8db7b2b20aba5873
I20251024 19:48:18.456799 1305 leader_election.cc:304] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:18.457278 2936 raft_consensus.cc:2749] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:48:18.460922 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 2851d58164d64370ba1f11be29fa1d33 (DEFAULT_TABLE table=default.Table_088895c10fa249e488c6cd194ea431fd [id=d0fd368e72a14c58a4f9e9d08b4ee649]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.461542 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2851d58164d64370ba1f11be29fa1d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.461859 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 129debb55f4a4b7295a8ad88e55ebc83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.464852 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 2842f8a6e04d4904bab4e9236325614c (DEFAULT_TABLE table=default.Table_32da551817ca47239b06d0c761a20839 [id=8698417054e043b1838b41aab19cd6a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.465245 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 2851d58164d64370ba1f11be29fa1d33 (DEFAULT_TABLE table=default.Table_088895c10fa249e488c6cd194ea431fd [id=d0fd368e72a14c58a4f9e9d08b4ee649]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.465842 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2842f8a6e04d4904bab4e9236325614c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.466045 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2851d58164d64370ba1f11be29fa1d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.474822 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 6a93c672295a40e5a79501c0ff3d1a3c (DEFAULT_TABLE table=default.Table_5f488d12a77948e68c3b5c1152e003f7 [id=5ffa582758154e6694adf569169c096b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.475426 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a93c672295a40e5a79501c0ff3d1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.478389 1774 ts_tablet_manager.cc:1929] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:18.478890 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 4ddce416119e4f08b81be4b5e6760144 (DEFAULT_TABLE table=default.Table_15b382636dbe430c991bff2369719fc7 [id=cf390d9bc2b24f8fa17f98760e93075b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.479338 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ddce416119e4f08b81be4b5e6760144. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.478927 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 6a93c672295a40e5a79501c0ff3d1a3c (DEFAULT_TABLE table=default.Table_5f488d12a77948e68c3b5c1152e003f7 [id=5ffa582758154e6694adf569169c096b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.481199 1483 tablet_service.cc:1505] Processing CreateTablet for tablet c9761b71a7294c78bccb5be86529dbe8 (DEFAULT_TABLE table=default.Table_9c8900b6f68e44cbb3333b6ffcb23eac [id=54e44ecda7a347d58070fa000c0b7bf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.481652 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9761b71a7294c78bccb5be86529dbe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.483551 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a93c672295a40e5a79501c0ff3d1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.488306 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 4ddce416119e4f08b81be4b5e6760144 (DEFAULT_TABLE table=default.Table_15b382636dbe430c991bff2369719fc7 [id=cf390d9bc2b24f8fa17f98760e93075b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.488778 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ddce416119e4f08b81be4b5e6760144. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.489737 1774 log.cc:1199] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/07e75c7f35544bec82cb994dea5ee4ef
I20251024 19:48:18.490409 1774 ts_tablet_manager.cc:1950] T 07e75c7f35544bec82cb994dea5ee4ef P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:18.491050 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 754) took 4170 ms. Trace:
I20251024 19:48:18.491251 1774 rpcz_store.cc:276] 1024 19:48:14.320209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.321016 (+ 807us) service_pool.cc:225] Handling call
1024 19:48:18.357517 (+4036501us) ts_tablet_manager.cc:1151] Deleting tablet 07e75c7f35544bec82cb994dea5ee4ef
1024 19:48:18.357528 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.375230 (+ 17702us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x616000163730. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f9890e171b2 00007f98939d3287 00007f9893a062d4 00007f98939e6d70 00007f98939e6afd 0000000000a8867c 00007f9883c49d7f 00007f9883c4d04b 00007f9883c4cec8 0000000000a8867c
1024 19:48:18.382394 (+ 7164us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.478274 (+ 95880us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.491010 (+ 12736us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6650112,"tablet-delete.queue_time_us":4036345}
I20251024 19:48:18.492413 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 755) took 4172 ms. Trace:
I20251024 19:48:18.492632 1774 rpcz_store.cc:276] 1024 19:48:14.320311 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.321196 (+ 885us) service_pool.cc:225] Handling call
1024 19:48:18.492195 (+4170999us) ts_tablet_manager.cc:1151] Deleting tablet b74415cf7c7e47a696b454a5d44d8017
1024 19:48:18.492209 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.492394 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4170826}
I20251024 19:48:18.493444 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 757) took 4172 ms. Trace:
I20251024 19:48:18.493675 1774 rpcz_store.cc:276] 1024 19:48:14.320532 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.320746 (+ 214us) service_pool.cc:225] Handling call
1024 19:48:18.493209 (+4172463us) ts_tablet_manager.cc:1151] Deleting tablet cf18f375266b4448b3f67afd787df6ad
1024 19:48:18.493248 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.493400 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4171300}
I20251024 19:48:18.508801 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 758) took 4186 ms. Trace:
I20251024 19:48:18.510613 1474 tablet_service.cc:1505] Processing CreateTablet for tablet ca344497ac094dd2aa4d0e361d6f518c (DEFAULT_TABLE table=default.Table_33ccbd1889d74b2e909e6ede2f837ef1 [id=e2273f51bd48478b8b6fb265089af985]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.510759 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 07e75c7f35544bec82cb994dea5ee4ef (table default.Table_83b4409228694c939a4a734c3b6bdd76 [id=450dc07c78d343aea5c8c2d2a6e26444]) successfully deleted
W20251024 19:48:18.511024 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b74415cf7c7e47a696b454a5d44d8017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b74415cf7c7e47a696b454a5d44d8017 already in progress: creating tablet
I20251024 19:48:18.511114 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca344497ac094dd2aa4d0e361d6f518c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:18.511397 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet cf18f375266b4448b3f67afd787df6ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf18f375266b4448b3f67afd787df6ad already in progress: creating tablet
I20251024 19:48:18.509294 1774 rpcz_store.cc:276] 1024 19:48:14.321815 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.322593 (+ 778us) service_pool.cc:225] Handling call
1024 19:48:18.494479 (+4171886us) ts_tablet_manager.cc:1151] Deleting tablet 839cef50795840fab564b94d88565f8c
1024 19:48:18.494512 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.508738 (+ 14226us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4171716}
I20251024 19:48:18.512809 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 767) took 3855 ms. Trace:
I20251024 19:48:18.512959 1774 rpcz_store.cc:276] 1024 19:48:14.657483 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.657558 (+ 75us) service_pool.cc:225] Handling call
1024 19:48:18.512461 (+3854903us) ts_tablet_manager.cc:1151] Deleting tablet b3e9960b7821421aa23e7ef5389f894a
1024 19:48:18.512502 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.512763 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3854666}
I20251024 19:48:18.515523 1341 tablet_service.cc:1505] Processing CreateTablet for tablet ca344497ac094dd2aa4d0e361d6f518c (DEFAULT_TABLE table=default.Table_33ccbd1889d74b2e909e6ede2f837ef1 [id=e2273f51bd48478b8b6fb265089af985]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.516165 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca344497ac094dd2aa4d0e361d6f518c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.518373 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 2842f8a6e04d4904bab4e9236325614c (DEFAULT_TABLE table=default.Table_32da551817ca47239b06d0c761a20839 [id=8698417054e043b1838b41aab19cd6a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.518749 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2842f8a6e04d4904bab4e9236325614c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.520696 2926 raft_consensus.cc:2804] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:18.520787 2926 raft_consensus.cc:493] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:18.520852 2926 raft_consensus.cc:3060] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.523720 2926 raft_consensus.cc:515] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.524431 2926 leader_election.cc:290] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:18.524906 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 160452us
I20251024 19:48:18.525170 1586 leader_election.cc:304] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:18.391913 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 977b4070419948d4910e2b69e01ff132 (DEFAULT_TABLE table=default.Table_fe56cf0299064493aa6b4f993e0ad929 [id=a860277e104846ac8cec903d14a7020c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.525712 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 977b4070419948d4910e2b69e01ff132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.528158 2936 raft_consensus.cc:493] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10a9536207e343f188e0ef709081d04e)
I20251024 19:48:18.392241 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 2869d5a7a29d479d891c3a4ba50e1ff2 (DEFAULT_TABLE table=default.Table_468b839284304d4eb9920feeeb49bc18 [id=8e3f7bc3fff24aa1a663cbc6cdb8d08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.528416 2936 raft_consensus.cc:515] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.399991 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 36ccb0cfba7a4a14b36d38897f4a9ef0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a94c856db4141cfb7ada7515d5b6e9e) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.404264 1617 tablet_service.cc:1505] Processing CreateTablet for tablet c8a2b5a84ce24935a06c8e8d7a70cb59 (DEFAULT_TABLE table=default.Table_dbb22386d4e04757bc3b5e73e46c6e1a [id=6eb631a1bb944115b56c51257a0bb716]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.528874 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8a2b5a84ce24935a06c8e8d7a70cb59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.528967 1623 tablet_service.cc:1505] Processing CreateTablet for tablet ca344497ac094dd2aa4d0e361d6f518c (DEFAULT_TABLE table=default.Table_33ccbd1889d74b2e909e6ede2f837ef1 [id=e2273f51bd48478b8b6fb265089af985]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.529181 2936 leader_election.cc:290] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:18.529290 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca344497ac094dd2aa4d0e361d6f518c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:18.530313 1308 leader_election.cc:343] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 9b89a90bb4ca447fbaf34fe39001c261
W20251024 19:48:18.531001 1305 leader_election.cc:343] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 9b89a90bb4ca447fbaf34fe39001c261
I20251024 19:48:18.531107 1305 leader_election.cc:304] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:18.531669 2936 raft_consensus.cc:2749] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:48:18.404469 1611 tablet_service.cc:1505] Processing CreateTablet for tablet f2fe735d85414bcda79ea4399adc041f (DEFAULT_TABLE table=default.Table_d2f10524d22141078abbeeea11a129ef [id=5354680146fb4514a83484157f46a1fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.532197 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2fe735d85414bcda79ea4399adc041f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.513854 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 769) took 3827 ms. Trace:
I20251024 19:48:18.537144 1774 rpcz_store.cc:276] 1024 19:48:14.686153 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.686214 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:18.513536 (+3827322us) ts_tablet_manager.cc:1151] Deleting tablet e51e597b8fd044e393a3af07da25586c
1024 19:48:18.513547 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.513729 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3827097}
I20251024 19:48:18.404264 1629 tablet_service.cc:1505] Processing CreateTablet for tablet f716f8585d4447629262089e6d7cefc9 (DEFAULT_TABLE table=default.Table_77d2dbff62c647e2b909bc059207d895 [id=1fc5cb1865c6446ebef50b6b74ab6359]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.537817 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f716f8585d4447629262089e6d7cefc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.537801 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 771) took 3848 ms. Trace:
I20251024 19:48:18.537894 1774 rpcz_store.cc:276] 1024 19:48:14.689356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.689410 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:18.537599 (+3848189us) ts_tablet_manager.cc:1151] Deleting tablet 9684072368f14cc2942b4e240f90b0f2
1024 19:48:18.537612 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.537759 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3847943}
I20251024 19:48:18.538590 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 772) took 3820 ms. Trace:
I20251024 19:48:18.538707 1774 rpcz_store.cc:276] 1024 19:48:14.718504 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.719333 (+ 829us) service_pool.cc:225] Handling call
1024 19:48:18.538391 (+3819058us) ts_tablet_manager.cc:1151] Deleting tablet 7cab9c17496947efbbe3cd9e5d8a49a8
1024 19:48:18.538400 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.538575 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3818796}
I20251024 19:48:18.539319 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 773) took 3817 ms. Trace:
I20251024 19:48:18.405643 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 6d1172e69d78456dbdbf0dba34d80db7 (DEFAULT_TABLE table=default.Table_303e87d506b64ecdbb134e0476487dec [id=aaaec8f080314033858bc8320c10cab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.541966 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d1172e69d78456dbdbf0dba34d80db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.405853 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet b95989b98c77472faa7bf940cfa60432 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.406134 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 995abf5a55b7445999c2b2c161610019 (DEFAULT_TABLE table=default.Table_658377c2e7954211a0e56c012dc08bf6 [id=0a1b6e5e13b340a69060753aa21590c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.545958 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 995abf5a55b7445999c2b2c161610019. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.539513 1774 rpcz_store.cc:276] 1024 19:48:14.722015 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.722091 (+ 76us) service_pool.cc:225] Handling call
1024 19:48:18.539182 (+3817091us) ts_tablet_manager.cc:1151] Deleting tablet 292c31e7d4ce411bb336a1405ca43ba5
1024 19:48:18.539190 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.539304 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3816858}
I20251024 19:48:18.409785 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 96e8588e092f45c2b10f4e1e97408c47 (DEFAULT_TABLE table=default.Table_6f21add4230f4d1692ce71616d05ff7a [id=55f8fb6f42cb41ca8aa74748a992413d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.410154 1627 tablet_service.cc:1505] Processing CreateTablet for tablet 4ec69ffe2c024d1a8596a93183d1458b (DEFAULT_TABLE table=default.Table_9ff460fc04ce46c0a4ae4163efee611b [id=c748eed9ebab4b7c9f43f61c12419808]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.549842 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec69ffe2c024d1a8596a93183d1458b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.550212 1774 tablet_replica.cc:333] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:18.552050 1774 raft_consensus.cc:2243] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.410473 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 366c17dbe45a4f8f9830dca68f621121 (DEFAULT_TABLE table=default.Table_6233280a74da4d1981019506fd1be2f3 [id=c6fd443442b2458ea8842a1ec8b67081]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.553704 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 366c17dbe45a4f8f9830dca68f621121. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.553259 1774 raft_consensus.cc:2272] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.410784 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 8ecb196eba1e4317af472ef753894bb9 (DEFAULT_TABLE table=default.Table_88a506f2737145189da07becb2912c29 [id=0fda2b1307fe40d1923326929f2bee2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.557428 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ecb196eba1e4317af472ef753894bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.411087 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 2fb5f8d81a924240a53715f6a43ab87f (DEFAULT_TABLE table=default.Table_2acb8ac85b4e4eae9ec2e97990416a3f [id=15352b76444849e2a89000475e102023]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.558532 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fb5f8d81a924240a53715f6a43ab87f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.558615 1774 ts_tablet_manager.cc:1916] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.443151 1624 tablet_service.cc:1505] Processing CreateTablet for tablet a9b1f6e4597045bfbe4a4645f105d8c8 (DEFAULT_TABLE table=default.Table_074163dbc16146cf81fc6db79bd636b0 [id=7f1c07c184f84adcba5735f76bdf98a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.444036 1776 ts_tablet_manager.cc:1929] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:18.561721 1776 log.cc:1199] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/e29abfa3bc8944aea7d1acd03e3eb897
I20251024 19:48:18.561897 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9b1f6e4597045bfbe4a4645f105d8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.562139 1776 ts_tablet_manager.cc:1950] T e29abfa3bc8944aea7d1acd03e3eb897 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:18.459681 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 2851d58164d64370ba1f11be29fa1d33 (DEFAULT_TABLE table=default.Table_088895c10fa249e488c6cd194ea431fd [id=d0fd368e72a14c58a4f9e9d08b4ee649]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.562480 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2851d58164d64370ba1f11be29fa1d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.565420 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e8588e092f45c2b10f4e1e97408c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.460465 1626 tablet_service.cc:1505] Processing CreateTablet for tablet 2842f8a6e04d4904bab4e9236325614c (DEFAULT_TABLE table=default.Table_32da551817ca47239b06d0c761a20839 [id=8698417054e043b1838b41aab19cd6a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.466529 1628 tablet_service.cc:1505] Processing CreateTablet for tablet c9761b71a7294c78bccb5be86529dbe8 (DEFAULT_TABLE table=default.Table_9c8900b6f68e44cbb3333b6ffcb23eac [id=54e44ecda7a347d58070fa000c0b7bf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.565951 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2842f8a6e04d4904bab4e9236325614c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.472139 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 4ddce416119e4f08b81be4b5e6760144 (DEFAULT_TABLE table=default.Table_15b382636dbe430c991bff2369719fc7 [id=cf390d9bc2b24f8fa17f98760e93075b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.566217 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ddce416119e4f08b81be4b5e6760144. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.477662 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 6a93c672295a40e5a79501c0ff3d1a3c (DEFAULT_TABLE table=default.Table_5f488d12a77948e68c3b5c1152e003f7 [id=5ffa582758154e6694adf569169c096b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.569763 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a93c672295a40e5a79501c0ff3d1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.573706 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9761b71a7294c78bccb5be86529dbe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.595796 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 920) took 1332 ms. Trace:
I20251024 19:48:18.595911 1776 rpcz_store.cc:276] 1024 19:48:17.263450 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.263540 (+ 90us) service_pool.cc:225] Handling call
1024 19:48:18.216615 (+953075us) ts_tablet_manager.cc:1151] Deleting tablet e29abfa3bc8944aea7d1acd03e3eb897
1024 19:48:18.216620 (+ 5us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.345330 (+128710us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.443940 (+ 98610us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.595769 (+151829us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":951537}
I20251024 19:48:18.596276 1776 tablet_replica.cc:333] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:18.596452 1776 raft_consensus.cc:2243] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.607710 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2869d5a7a29d479d891c3a4ba50e1ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.616777 2936 raft_consensus.cc:493] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.616958 2936 raft_consensus.cc:515] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.617674 2936 leader_election.cc:290] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:18.618499 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9489a01434a24d9ca94791252950725c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:18.618745 1507 raft_consensus.cc:2393] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:18.630107 1774 ts_tablet_manager.cc:1929] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:18.631363 1774 log.cc:1199] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/27b32126d04e480f8db7b2b20aba5873
I20251024 19:48:18.635210 1774 ts_tablet_manager.cc:1950] T 27b32126d04e480f8db7b2b20aba5873 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:18.639326 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 774) took 3875 ms. Trace:
I20251024 19:48:18.640043 1774 rpcz_store.cc:276] 1024 19:48:14.763984 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.764149 (+ 165us) service_pool.cc:225] Handling call
1024 19:48:18.549735 (+3785586us) ts_tablet_manager.cc:1151] Deleting tablet 27b32126d04e480f8db7b2b20aba5873
1024 19:48:18.549917 (+ 182us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.595464 (+ 45547us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.628947 (+ 33483us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.639132 (+ 10185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3784955}
I20251024 19:48:18.644239 1774 tablet_replica.cc:333] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:18.644447 1766 tablet_bootstrap.cc:700] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e: Time spent opening tablet: real 0.262s user 0.000s sys 0.000s
I20251024 19:48:18.644850 1766 tablet_bootstrap.cc:654] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.508605 1767 tablet_bootstrap.cc:492] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:18.647774 1767 ts_tablet_manager.cc:1403] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.378s user 0.004s sys 0.000s
I20251024 19:48:18.647410 1774 raft_consensus.cc:2243] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.649134 1774 raft_consensus.cc:2272] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.649865 2926 raft_consensus.cc:2804] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:18.649992 2926 raft_consensus.cc:493] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:18.650125 2926 raft_consensus.cc:3060] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.651122 1776 raft_consensus.cc:2272] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.651436 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9489a01434a24d9ca94791252950725c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:18.651686 1776 ts_tablet_manager.cc:1916] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.653061 2926 raft_consensus.cc:515] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.653630 1305 leader_election.cc:304] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:18.653672 2926 leader_election.cc:290] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:18.654054 2936 raft_consensus.cc:3060] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.656878 2936 raft_consensus.cc:2749] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:18.658565 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 133424us
I20251024 19:48:18.659449 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94fc8c5716af47faa4e831f05fce8610" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:18.659662 1507 raft_consensus.cc:3060] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.660894 1776 ts_tablet_manager.cc:1929] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:18.660988 1776 log.cc:1199] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/c82f60819e7a442bbc0cc04659e14e87
I20251024 19:48:18.661365 1776 ts_tablet_manager.cc:1950] T c82f60819e7a442bbc0cc04659e14e87 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:18.662936 1507 raft_consensus.cc:2468] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:18.674247 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 922) took 1393 ms. Trace:
I20251024 19:48:18.674759 1776 rpcz_store.cc:276] 1024 19:48:17.280275 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.280781 (+ 506us) service_pool.cc:225] Handling call
1024 19:48:18.596227 (+1315446us) ts_tablet_manager.cc:1151] Deleting tablet c82f60819e7a442bbc0cc04659e14e87
1024 19:48:18.596236 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.655305 (+ 59069us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.660805 (+ 5500us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.674215 (+ 13410us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54486,"tablet-delete.queue_time_us":1315331}
I20251024 19:48:18.676294 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 925) took 1395 ms. Trace:
I20251024 19:48:18.680902 1776 rpcz_store.cc:276] 1024 19:48:17.280557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.280615 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:18.676167 (+1395552us) ts_tablet_manager.cc:1151] Deleting tablet 187bc968c2b049028dd25dfb0264c0aa
1024 19:48:18.676181 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.676275 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31,"tablet-delete.queue_time_us":1395263}
I20251024 19:48:18.681519 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62667092d7e244cca1020ec220ea9124" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:18.681874 1507 raft_consensus.cc:3060] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.682950 1776 tablet_replica.cc:333] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:18.683910 1776 raft_consensus.cc:2243] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.684573 1776 raft_consensus.cc:2272] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.685989 1507 raft_consensus.cc:2468] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:18.682178 1767 raft_consensus.cc:359] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.686173 1767 raft_consensus.cc:385] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.686445 1767 raft_consensus.cc:740] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.688246 1767 consensus_queue.cc:260] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.688292 1345 tablet_service.cc:1505] Processing CreateTablet for tablet c9761b71a7294c78bccb5be86529dbe8 (DEFAULT_TABLE table=default.Table_9c8900b6f68e44cbb3333b6ffcb23eac [id=54e44ecda7a347d58070fa000c0b7bf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.688953 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9761b71a7294c78bccb5be86529dbe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.691259 1767 ts_tablet_manager.cc:1434] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.043s user 0.000s sys 0.001s
I20251024 19:48:18.692449 1767 tablet_bootstrap.cc:492] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:18.695930 1767 tablet_bootstrap.cc:654] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.696986 3126 tablet.cc:1722] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:18.693629 3137 consensus_queue.cc:1048] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:18.698462 2936 raft_consensus.cc:493] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.698644 2936 raft_consensus.cc:515] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.687798 1776 ts_tablet_manager.cc:1916] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.699282 2936 leader_election.cc:290] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:18.699681 3127 tablet.cc:1722] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:18.699954 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2602335327fc489fa154056c5ef5138f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:18.710325 1766 tablet_bootstrap.cc:492] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:18.710707 1766 ts_tablet_manager.cc:1403] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.399s user 0.005s sys 0.000s
I20251024 19:48:18.711145 2936 raft_consensus.cc:493] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.711308 2936 raft_consensus.cc:515] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.711390 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f630fd412ca4c4f8570359c1678e6af" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:18.711572 1505 raft_consensus.cc:3060] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.711627 1766 raft_consensus.cc:359] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:18.711769 1766 raft_consensus.cc:385] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.711826 1766 raft_consensus.cc:740] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.711992 2936 leader_election.cc:290] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:18.712133 1766 consensus_queue.cc:260] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:18.712949 1766 ts_tablet_manager.cc:1434] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:18.713743 1774 ts_tablet_manager.cc:1916] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.714793 1505 raft_consensus.cc:2468] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:18.726377 1775 ts_tablet_manager.cc:1929] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:18.726608 1775 log.cc:1199] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/16bc737104ab46b3b071a0760dbc847d
W20251024 19:48:18.513810 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 839cef50795840fab564b94d88565f8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 839cef50795840fab564b94d88565f8c already in progress: creating tablet
W20251024 19:48:18.733242 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 219415us
W20251024 19:48:18.733656 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b3e9960b7821421aa23e7ef5389f894a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3e9960b7821421aa23e7ef5389f894a already in progress: creating tablet
W20251024 19:48:18.734326 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet e51e597b8fd044e393a3af07da25586c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e51e597b8fd044e393a3af07da25586c already in progress: creating tablet
W20251024 19:48:18.734613 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9684072368f14cc2942b4e240f90b0f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9684072368f14cc2942b4e240f90b0f2 already in progress: creating tablet
W20251024 19:48:18.734926 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7cab9c17496947efbbe3cd9e5d8a49a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7cab9c17496947efbbe3cd9e5d8a49a8 already in progress: creating tablet
W20251024 19:48:18.735200 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 292c31e7d4ce411bb336a1405ca43ba5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 292c31e7d4ce411bb336a1405ca43ba5 already in progress: creating tablet
I20251024 19:48:18.735463 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 27b32126d04e480f8db7b2b20aba5873 (table default.Table_a372d22e91854206b3523ecd087303ca [id=dfffb9501a0549198d17b197ef05871e]) successfully deleted
I20251024 19:48:18.651281 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet e29abfa3bc8944aea7d1acd03e3eb897 (table default.Table_55339e02660843c3a8536a4b7a06831e [id=ba30503223dd4979824ec9c1a50a8ae2]) successfully deleted
W20251024 19:48:18.736840 1176 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 85586.8us
I20251024 19:48:18.737561 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3b7821f2ba4a40a6a1d6c6ae777feb70" } modify_external_catalogs: true
I20251024 19:48:18.737843 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3b7821f2ba4a40a6a1d6c6ae777feb70" } modify_external_catalogs: true
I20251024 19:48:18.737818 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fe56cf0299064493aa6b4f993e0ad929" } new_table_name: "default.Table_84aeb3904e8f490c85eb6a322e05d144" modify_external_catalogs: true
I20251024 19:48:18.738523 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet c82f60819e7a442bbc0cc04659e14e87 (table default.Table_6f376570019346199a3c8c77d879fa1c [id=c5f7c2469f394bba8b91e2f20170963e]) successfully deleted
W20251024 19:48:18.738724 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 187bc968c2b049028dd25dfb0264c0aa because the tablet was not found. No further retry: Not found: Tablet not found: 187bc968c2b049028dd25dfb0264c0aa
I20251024 19:48:18.738687 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_81453b8401974cbcaa50e97a3c3bf12d" } new_table_name: "default.Table_da24358ca9544a0e805bb937715888f9" modify_external_catalogs: true
W20251024 19:48:18.690027 1178 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 93d84776848049c8a146998219986a84 (table default.Table_aea1b9672b8c4d1698e782c6824de441 [id=374823b8c35b46d8a9dba817f54822b8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 93d84776848049c8a146998219986a84 consensus state has no leader
I20251024 19:48:18.739461 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_effb50443d274effaa5039639c921715"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:18.741837 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_68e3a1ca444149a2ad1ce7c1d54ad12b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:18.742708 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7d5f945cec1f41f2abd299341bb4fff2" } new_table_name: "default.Table_94fcb02d0c374222a5ceb6163a6b9f2f" modify_external_catalogs: true
I20251024 19:48:18.742743 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5783546824a846e7b084ac7e809885b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:18.745975 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b9cb3ccac55f4b0a8897f5cd1412440d" } modify_external_catalogs: true
I20251024 19:48:18.746156 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b9cb3ccac55f4b0a8897f5cd1412440d" } modify_external_catalogs: true
W20251024 19:48:18.748129 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5783546824a846e7b084ac7e809885b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:18.749264 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68e3a1ca444149a2ad1ce7c1d54ad12b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.749352 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9c8900b6f68e44cbb3333b6ffcb23eac" } modify_external_catalogs: true
W20251024 19:48:18.749927 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_effb50443d274effaa5039639c921715 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.749886 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9c8900b6f68e44cbb3333b6ffcb23eac" } modify_external_catalogs: true
W20251024 19:48:18.765985 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 16a8c772ac404ae2b2f86d948a0fcd2f consensus state has no leader
W20251024 19:48:18.766294 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4100abc9ff12471e9a4959b6b24f7e99 consensus state has no leader
W20251024 19:48:18.766371 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 977b4070419948d4910e2b69e01ff132 consensus state has no leader
I20251024 19:48:18.766746 1776 ts_tablet_manager.cc:1929] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:18.766880 1776 log.cc:1199] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/5d129bdc876e425aabcf1c99997e319a
I20251024 19:48:18.767832 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 54bc7c7764584e11948d2831a4e5e0b6
I20251024 19:48:18.771234 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 54bc7c7764584e11948d2831a4e5e0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.773932 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_532ecd1242ea40d29a9d1bcf6fa8a488" } modify_external_catalogs: true
I20251024 19:48:18.774006 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ea05abbf4e984c73a9e8df1fa0a505a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:18.774142 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_532ecd1242ea40d29a9d1bcf6fa8a488" } modify_external_catalogs: true
W20251024 19:48:18.774545 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea05abbf4e984c73a9e8df1fa0a505a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.775825 1774 ts_tablet_manager.cc:1929] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:18.775995 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_bcb7bf7c40d44f56afff3157570ddf05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:18.776786 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcb7bf7c40d44f56afff3157570ddf05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.777637 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a9bb007e715a4e80b9f3c77192d69c80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:18.778133 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9bb007e715a4e80b9f3c77192d69c80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.752883 1766 tablet_bootstrap.cc:492] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:18.780987 1766 tablet_bootstrap.cc:654] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.783697 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4193750331874d7ba9bbe6279c3a8e37" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:18.784967 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 81382.7us
I20251024 19:48:18.786386 1774 log.cc:1199] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/9b89a90bb4ca447fbaf34fe39001c261
I20251024 19:48:18.786954 1774 ts_tablet_manager.cc:1950] T 9b89a90bb4ca447fbaf34fe39001c261 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:18.787446 1776 ts_tablet_manager.cc:1950] T 5d129bdc876e425aabcf1c99997e319a P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:18.787932 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 775) took 4023 ms. Trace:
I20251024 19:48:18.788067 1774 rpcz_store.cc:276] 1024 19:48:14.764093 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.764163 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:18.643464 (+3879301us) ts_tablet_manager.cc:1151] Deleting tablet 9b89a90bb4ca447fbaf34fe39001c261
1024 19:48:18.643479 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.765085 (+121606us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.775713 (+ 10628us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.787911 (+ 12198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3877728}
I20251024 19:48:18.788211 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 923) took 1507 ms. Trace:
I20251024 19:48:18.788304 1776 rpcz_store.cc:276] 1024 19:48:17.280366 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.280939 (+ 573us) service_pool.cc:225] Handling call
1024 19:48:18.682892 (+1401953us) ts_tablet_manager.cc:1151] Deleting tablet 5d129bdc876e425aabcf1c99997e319a
1024 19:48:18.682905 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.756937 (+ 74032us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.766627 (+ 9690us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.788195 (+ 21568us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1401837}
I20251024 19:48:18.788616 1776 tablet_replica.cc:333] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:18.788815 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 5d129bdc876e425aabcf1c99997e319a (table default.Table_a4b63032d2224a6d95765cb93137f4a2 [id=d28fd80220c64f8496a09f5d4e712874]) successfully deleted
I20251024 19:48:18.788856 1776 raft_consensus.cc:2243] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:18.788898 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 777) took 4003 ms. Trace:
I20251024 19:48:18.788962 1774 rpcz_store.cc:276] 1024 19:48:14.785253 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.785311 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:18.788772 (+4003461us) ts_tablet_manager.cc:1151] Deleting tablet 520060a9adca4bb0b8616bcc006dce52
1024 19:48:18.788784 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.788883 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4003281}
I20251024 19:48:18.789120 1776 raft_consensus.cc:2272] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.789354 1774 tablet_replica.cc:333] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:18.793300 1768 tablet_bootstrap.cc:492] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:18.793358 2923 raft_consensus.cc:493] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.793464 1768 ts_tablet_manager.cc:1403] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.448s user 0.030s sys 0.108s
I20251024 19:48:18.793663 2923 raft_consensus.cc:515] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.788223 1586 leader_election.cc:304] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:18.789588 1774 raft_consensus.cc:2243] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.794471 1774 raft_consensus.cc:2272] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.794667 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet de2ebb79d4fe4a1bbac372bb26f370ed
I20251024 19:48:18.795153 1774 ts_tablet_manager.cc:1916] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.795514 3138 raft_consensus.cc:2804] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:18.796249 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_243a93945dd5416e94a881c0e85e5875"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:18.796371 1586 leader_election.cc:304] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:18.796741 2926 raft_consensus.cc:2804] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:18.797179 1586 leader_election.cc:304] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:18.797199 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_243a93945dd5416e94a881c0e85e5875 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.797376 3137 raft_consensus.cc:2804] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:18.793258 1776 ts_tablet_manager.cc:1916] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.796001 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet de2ebb79d4fe4a1bbac372bb26f370ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.801218 1768 raft_consensus.cc:359] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.801671 1768 raft_consensus.cc:385] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.797482 3137 raft_consensus.cc:697] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:18.802435 1766 tablet_bootstrap.cc:492] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:18.802526 1768 raft_consensus.cc:740] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.802728 1766 ts_tablet_manager.cc:1403] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.004s
I20251024 19:48:18.802800 1768 consensus_queue.cc:260] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.803820 3138 raft_consensus.cc:697] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:18.803993 2926 raft_consensus.cc:697] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:18.803747 1768 ts_tablet_manager.cc:1434] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20251024 19:48:18.805770 1418 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:18.806823 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2602335327fc489fa154056c5ef5138f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:18.807031 1508 raft_consensus.cc:2393] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:18.807132 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4193750331874d7ba9bbe6279c3a8e37" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:18.807834 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94fc8c5716af47faa4e831f05fce8610" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:18.809197 1768 tablet_bootstrap.cc:492] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:18.810892 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62667092d7e244cca1020ec220ea9124" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:18.811031 1768 tablet_bootstrap.cc:654] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.811532 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f630fd412ca4c4f8570359c1678e6af" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:18.812204 1766 raft_consensus.cc:359] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:18.812462 1766 raft_consensus.cc:385] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.812564 1766 raft_consensus.cc:740] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.812881 1766 consensus_queue.cc:260] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:18.813633 1766 ts_tablet_manager.cc:1434] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251024 19:48:18.814025 1766 tablet_bootstrap.cc:492] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:18.814103 1364 raft_consensus.cc:3060] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.814191 1774 ts_tablet_manager.cc:1929] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:18.815485 1774 log.cc:1199] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/b88a6a5237d146f5b6f082ca3af25b2b
I20251024 19:48:18.815995 1774 ts_tablet_manager.cc:1950] T b88a6a5237d146f5b6f082ca3af25b2b P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:18.816890 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 779) took 3902 ms. Trace:
I20251024 19:48:18.802160 3137 consensus_queue.cc:237] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.804188 2926 consensus_queue.cc:237] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.819244 1766 tablet_bootstrap.cc:654] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.819270 2923 leader_election.cc:290] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:18.819373 1774 rpcz_store.cc:276] 1024 19:48:14.914160 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.914229 (+ 69us) service_pool.cc:225] Handling call
1024 19:48:18.789310 (+3875081us) ts_tablet_manager.cc:1151] Deleting tablet b88a6a5237d146f5b6f082ca3af25b2b
1024 19:48:18.789319 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.804461 (+ 15142us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.814082 (+ 9621us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.816868 (+ 2786us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3874818}
I20251024 19:48:18.820190 1774 tablet_replica.cc:333] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:18.820466 1774 raft_consensus.cc:2243] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.820662 1364 raft_consensus.cc:3060] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.820730 1774 raft_consensus.cc:2272] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.821477 1774 ts_tablet_manager.cc:1916] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.827572 1308 leader_election.cc:304] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:18.831091 2936 raft_consensus.cc:3060] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.832872 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet c9761b71a7294c78bccb5be86529dbe8
I20251024 19:48:18.833750 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6d8df15af842462fb931f649dc0dd806" } new_table_name: "default.Table_3cebb5cdb61a4ca0a6511d5157172859" modify_external_catalogs: true
I20251024 19:48:18.834081 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1cf2ee7da8ea49c399e3e7cbb4592c8f" } new_table_name: "default.Table_d494ae379f1a42cabbe4ea77a4668e45" modify_external_catalogs: true
I20251024 19:48:18.834349 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ec77ed043045411b85ba06840af39907" } modify_external_catalogs: true
I20251024 19:48:18.835093 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet c9761b71a7294c78bccb5be86529dbe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.835371 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ec77ed043045411b85ba06840af39907" } modify_external_catalogs: true
I20251024 19:48:18.835399 1183 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8] alter complete (version 2)
I20251024 19:48:18.835515 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_291336b400e94ac188244241309455d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:18.836028 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_291336b400e94ac188244241309455d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:18.836766 2936 raft_consensus.cc:2749] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:18.837424 1308 leader_election.cc:304] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:18.837826 2936 raft_consensus.cc:2749] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:18.727002 1775 ts_tablet_manager.cc:1950] T 16bc737104ab46b3b071a0760dbc847d P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:18.841835 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 837) took 3785 ms. Trace:
I20251024 19:48:18.841981 1775 rpcz_store.cc:276] 1024 19:48:15.056357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.056421 (+ 64us) service_pool.cc:225] Handling call
1024 19:48:18.307295 (+3250874us) ts_tablet_manager.cc:1151] Deleting tablet 16bc737104ab46b3b071a0760dbc847d
1024 19:48:18.307306 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.496894 (+189588us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.726256 (+229362us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.841813 (+115557us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3250659}
I20251024 19:48:18.842542 1775 tablet_replica.cc:333] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:18.842803 1775 raft_consensus.cc:2243] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.842994 1775 raft_consensus.cc:2272] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.843554 1775 ts_tablet_manager.cc:1916] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.846125 1774 ts_tablet_manager.cc:1929] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:18.848392 1774 log.cc:1199] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/701c93a600964c20a275f5942a2ff863
I20251024 19:48:18.848903 1774 ts_tablet_manager.cc:1950] T 701c93a600964c20a275f5942a2ff863 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:18.849326 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 780) took 3867 ms. Trace:
I20251024 19:48:18.849465 1774 rpcz_store.cc:276] 1024 19:48:14.981504 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:14.982034 (+ 530us) service_pool.cc:225] Handling call
1024 19:48:18.820125 (+3838091us) ts_tablet_manager.cc:1151] Deleting tablet 701c93a600964c20a275f5942a2ff863
1024 19:48:18.820139 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.829697 (+ 9558us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.846019 (+ 16322us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.849310 (+ 3291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3837869}
W20251024 19:48:18.850400 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2ab00d07d27457882a9f35cc58eb5aa consensus state has no leader
I20251024 19:48:18.850502 1774 tablet_replica.cc:333] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:18.851037 1774 raft_consensus.cc:2243] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.851804 1774 raft_consensus.cc:2272] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:18.855304 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4229bd266e04d04b970bc9fc760155f consensus state has no leader
I20251024 19:48:18.863986 1774 ts_tablet_manager.cc:1916] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:18.864908 1586 leader_election.cc:343] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.865243 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067631b128fd4a1e9ab72c9507dab6c4" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:18.865542 1507 raft_consensus.cc:2468] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:18.868907 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067631b128fd4a1e9ab72c9507dab6c4" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:18.885444 1774 ts_tablet_manager.cc:1929] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:18.885878 1774 log.cc:1199] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/4243f11bf2b34b30b10766c19e2c6409
I20251024 19:48:18.886613 1774 ts_tablet_manager.cc:1950] T 4243f11bf2b34b30b10766c19e2c6409 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:18.887878 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 781) took 3835 ms. Trace:
I20251024 19:48:18.888892 1774 rpcz_store.cc:276] 1024 19:48:15.052733 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.054211 (+ 1478us) service_pool.cc:225] Handling call
1024 19:48:18.850455 (+3796244us) ts_tablet_manager.cc:1151] Deleting tablet 4243f11bf2b34b30b10766c19e2c6409
1024 19:48:18.850465 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.870848 (+ 20383us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.883720 (+ 12872us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.887856 (+ 4136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3796010}
I20251024 19:48:18.889755 1774 tablet_replica.cc:333] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:18.893635 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 54bc7c7764584e11948d2831a4e5e0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.894025 1774 raft_consensus.cc:2243] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.894593 1774 raft_consensus.cc:2272] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.894883 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet de2ebb79d4fe4a1bbac372bb26f370ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:18.895431 1774 ts_tablet_manager.cc:1916] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.895591 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet de2ebb79d4fe4a1bbac372bb26f370ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.818920 3138 consensus_queue.cc:237] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.884461 3080 raft_consensus.cc:493] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.895982 3080 raft_consensus.cc:515] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.896682 3080 leader_election.cc:290] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:18.891548 1767 tablet_bootstrap.cc:492] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:18.897120 1767 ts_tablet_manager.cc:1403] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.205s user 0.000s sys 0.016s
I20251024 19:48:18.898797 1768 tablet_bootstrap.cc:492] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:18.898964 1768 ts_tablet_manager.cc:1403] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.090s user 0.007s sys 0.000s
I20251024 19:48:18.899721 1768 raft_consensus.cc:359] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.899916 1768 raft_consensus.cc:385] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.899976 1768 raft_consensus.cc:740] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.900168 1768 consensus_queue.cc:260] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.900620 1768 ts_tablet_manager.cc:1434] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:18.900895 1768 tablet_bootstrap.cc:492] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:18.901165 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 54bc7c7764584e11948d2831a4e5e0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:18.901423 3080 consensus_queue.cc:1048] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:18.902176 1586 consensus_peers.cc:597] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:18.902623 1768 tablet_bootstrap.cc:654] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.903393 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet c9761b71a7294c78bccb5be86529dbe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:18.903970 1586 leader_election.cc:343] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.910090 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet c9761b71a7294c78bccb5be86529dbe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:18.910705 1586 leader_election.cc:343] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.892604 1776 ts_tablet_manager.cc:1929] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:18.914727 1776 log.cc:1199] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/3887fc21726846279fecb02672b9bc07
I20251024 19:48:18.915206 1776 ts_tablet_manager.cc:1950] T 3887fc21726846279fecb02672b9bc07 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:18.915568 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 924) took 1635 ms. Trace:
I20251024 19:48:18.916605 3080 consensus_queue.cc:1048] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:18.919901 1776 rpcz_store.cc:276] 1024 19:48:17.280464 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.281072 (+ 608us) service_pool.cc:225] Handling call
1024 19:48:18.788567 (+1507495us) ts_tablet_manager.cc:1151] Deleting tablet 3887fc21726846279fecb02672b9bc07
1024 19:48:18.788578 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.887941 (+ 99363us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.892480 (+ 4539us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.915553 (+ 23073us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1507374}
I20251024 19:48:18.920625 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 3887fc21726846279fecb02672b9bc07 (table default.Table_e378095ddecc4dcf801a1568160c34b4 [id=450b69bd10cf4c629cc3f8dadc790312]) successfully deleted
I20251024 19:48:18.917171 1767 raft_consensus.cc:359] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.921093 1767 raft_consensus.cc:385] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.921223 1767 raft_consensus.cc:740] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.921602 1767 consensus_queue.cc:260] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.921993 1776 tablet_replica.cc:333] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:18.922338 1776 raft_consensus.cc:2243] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:18.923509 1767 ts_tablet_manager.cc:1434] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20251024 19:48:18.923817 1776 raft_consensus.cc:2272] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.924551 1776 ts_tablet_manager.cc:1916] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.929957 1774 ts_tablet_manager.cc:1929] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:18.931414 1767 tablet_bootstrap.cc:492] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:48:18.932749 1586 leader_election.cc:343] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:18.933948 1774 log.cc:1199] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/cef80e4d14bb4e719bceebf43041d079
I20251024 19:48:18.934657 1774 ts_tablet_manager.cc:1950] T cef80e4d14bb4e719bceebf43041d079 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:18.935847 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 782) took 3882 ms. Trace:
I20251024 19:48:18.936780 1774 rpcz_store.cc:276] 1024 19:48:15.052908 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.060672 (+ 7764us) service_pool.cc:225] Handling call
1024 19:48:18.889705 (+3829033us) ts_tablet_manager.cc:1151] Deleting tablet cef80e4d14bb4e719bceebf43041d079
1024 19:48:18.889715 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.908229 (+ 18514us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.929808 (+ 21579us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.935828 (+ 6020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3828712}
I20251024 19:48:18.937696 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 784) took 3770 ms. Trace:
I20251024 19:48:18.938724 1766 tablet_bootstrap.cc:492] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:18.938913 1766 ts_tablet_manager.cc:1403] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.000s
I20251024 19:48:18.939883 1766 raft_consensus.cc:359] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:18.940142 1766 raft_consensus.cc:385] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.940248 1766 raft_consensus.cc:740] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.940567 1766 consensus_queue.cc:260] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:18.941249 1766 ts_tablet_manager.cc:1434] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:18.942811 1190 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 067631b128fd4a1e9ab72c9507dab6c4 (table default.Table_814a6dad46d54945b70e652b7e0f0032 [id=3f394c59d82b44928beb22f42312a4a3]). Expected version 1 got 0
I20251024 19:48:18.950838 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 346cb0bb8a0d43f7a15b788a303a6bcf
I20251024 19:48:18.952523 1766 tablet_bootstrap.cc:492] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:18.954629 1774 rpcz_store.cc:276] 1024 19:48:15.167445 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.167513 (+ 68us) service_pool.cc:225] Handling call
1024 19:48:18.937541 (+3770028us) ts_tablet_manager.cc:1151] Deleting tablet 9f791083b4964ff9b0f5fc575478892c
1024 19:48:18.937551 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.937678 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3769821}
I20251024 19:48:18.957553 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 783) took 3790 ms. Trace:
I20251024 19:48:18.958137 1774 rpcz_store.cc:276] 1024 19:48:15.167297 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.169862 (+ 2565us) service_pool.cc:225] Handling call
1024 19:48:18.957417 (+3787555us) ts_tablet_manager.cc:1151] Deleting tablet 95f62010a24f432c84c89345f9b02b2f
1024 19:48:18.957429 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.957535 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3787385}
I20251024 19:48:18.958786 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet af918843c74940888c709d9a05aa6e0f
I20251024 19:48:18.958858 1774 tablet_replica.cc:333] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:18.959054 1774 raft_consensus.cc:2243] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:18.959233 1774 raft_consensus.cc:2272] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:18.959816 1774 ts_tablet_manager.cc:1916] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:18.960594 1766 tablet_bootstrap.cc:654] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.966877 2936 raft_consensus.cc:493] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:18.967020 2936 raft_consensus.cc:515] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.967633 2936 leader_election.cc:290] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:18.968443 1308 leader_election.cc:343] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 16bc737104ab46b3b071a0760dbc847d
I20251024 19:48:18.974228 2926 consensus_queue.cc:1048] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:18.974921 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet cb0405765c6146cbbb187af96a60be2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.975152 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet af918843c74940888c709d9a05aa6e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.975368 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16bc737104ab46b3b071a0760dbc847d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:18.975490 1624 tablet_service.cc:1505] Processing CreateTablet for tablet c36f0d063c9e498c96c83de1c4692745 (DEFAULT_TABLE table=default.Table_266203ab5d8d40979ab181d5715a4348 [id=733f3fdc70e44f4c838d04bf66b08c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.975692 1768 tablet_bootstrap.cc:492] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:18.975783 3126 tablet.cc:1717] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97: Skipping requested alter to schema version 2, tablet already version 2
I20251024 19:48:18.975841 1768 ts_tablet_manager.cc:1403] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.000s
I20251024 19:48:18.975871 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 1a7468b6f68046fa91ce076ffe3f97a5 (DEFAULT_TABLE table=default.Table_4d0d05989b3844cd8813c527af4862a1 [id=98b4201eaf8a4d059f67fb980cdced3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.976102 1305 leader_election.cc:304] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:18.976218 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a7468b6f68046fa91ce076ffe3f97a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.976512 2936 raft_consensus.cc:2749] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:18.976516 1628 tablet_service.cc:1505] Processing CreateTablet for tablet cb1f850b3e104707a53d540ad5752d4d (DEFAULT_TABLE table=default.Table_12fe00934de64b41af4f5fa5980dd476 [id=ea3808eabb2a430a9313732c4a1d2c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.976673 1768 raft_consensus.cc:359] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.976816 1768 raft_consensus.cc:385] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:18.976868 1768 raft_consensus.cc:740] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:18.974238 1586 leader_election.cc:304] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:18.977069 1768 consensus_queue.cc:260] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:18.977346 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 346cb0bb8a0d43f7a15b788a303a6bcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:18.977592 1768 ts_tablet_manager.cc:1434] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:18.977890 1768 tablet_bootstrap.cc:492] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:18.981294 1618 tablet_service.cc:1505] Processing CreateTablet for tablet e927ba02569f42d2b6cd758f6e4d9113 (DEFAULT_TABLE table=default.Table_669783bf95a24c658906c5ea84b932b1 [id=231c6fbfbc4840428aaebf311b8f051e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.982087 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 90327823332a4a63a4e9e16fcf74563f (DEFAULT_TABLE table=default.Table_9423eab92cc74554b6a926d3c438fd30 [id=23639a33342f4a2396dc37d8905efc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.982959 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 97f6223d9e8b4f80b70b7515108d8492 (DEFAULT_TABLE table=default.Table_6cfafb2b827d4dba9de0de5bb5472304 [id=253d20af9be945b6a55bad0baf545bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:18.989627 1768 tablet_bootstrap.cc:654] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:18.992079 1191 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 067631b128fd4a1e9ab72c9507dab6c4 (table default.Table_814a6dad46d54945b70e652b7e0f0032 [id=3f394c59d82b44928beb22f42312a4a3]). Expected version 1 got 0
I20251024 19:48:18.992728 1774 ts_tablet_manager.cc:1929] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251024 19:48:18.992846 1774 log.cc:1199] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/8ba85164015448d4a688a473cce90c5c
I20251024 19:48:18.993217 1774 ts_tablet_manager.cc:1950] T 8ba85164015448d4a688a473cce90c5c P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:18.996659 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97f6223d9e8b4f80b70b7515108d8492. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:18.997066 2923 raft_consensus.cc:2804] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:18.997167 2923 raft_consensus.cc:493] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:18.997244 2923 raft_consensus.cc:3060] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:18.998675 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0405765c6146cbbb187af96a60be2e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:18.998888 1507 raft_consensus.cc:2468] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:18.999794 1366 raft_consensus.cc:3060] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.001237 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0405765c6146cbbb187af96a60be2e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:19.002017 2858 raft_consensus.cc:493] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.002149 2858 raft_consensus.cc:515] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.002247 3127 tablet.cc:1717] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e: Skipping requested alter to schema version 2, tablet already version 2
I20251024 19:48:19.002697 2858 leader_election.cc:290] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.003515 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 1fbbca3bad5f4b448ae5afc4d5cfed14 (DEFAULT_TABLE table=default.Table_33762464873048acb73f8d26e167301e [id=59b4d3aff42249eb8ff8f4292f690c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.005625 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d840888183ec4ef7bb245363c5bac322" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:19.006107 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 785) took 3734 ms. Trace:
W20251024 19:48:19.006225 1447 leader_election.cc:343] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:19.007368 1586 consensus_peers.cc:597] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:19.007562 1586 leader_election.cc:343] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.007508 1774 rpcz_store.cc:276] 1024 19:48:15.271902 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.273116 (+ 1214us) service_pool.cc:225] Handling call
1024 19:48:18.958805 (+3685689us) ts_tablet_manager.cc:1151] Deleting tablet 8ba85164015448d4a688a473cce90c5c
1024 19:48:18.958817 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.971615 (+ 12798us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:18.992607 (+ 20992us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.006086 (+ 13479us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3667764}
W20251024 19:48:19.007761 1586 consensus_peers.cc:597] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:19.008054 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 27729f3779454bf9bf7de8f7728ca1a6 (DEFAULT_TABLE table=default.Table_23a248d91f5d45fbbf4bf104efd8bfb2 [id=1863f145669846098f7568da3262dd3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.008204 3151 consensus_queue.cc:1048] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:19.008481 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d840888183ec4ef7bb245363c5bac322" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:19.009073 1444 leader_election.cc:343] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.009186 1444 leader_election.cc:304] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.009203 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 801) took 3490 ms. Trace:
I20251024 19:48:19.009267 1774 rpcz_store.cc:276] 1024 19:48:15.519189 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.531627 (+ 12438us) service_pool.cc:225] Handling call
1024 19:48:19.009088 (+3477461us) ts_tablet_manager.cc:1151] Deleting tablet 584cbe0c396841f794e665aae3ebbd91
1024 19:48:19.009099 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.009186 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3430703}
I20251024 19:48:19.009757 2857 raft_consensus.cc:2749] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:19.010217 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb1f850b3e104707a53d540ad5752d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.010466 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27729f3779454bf9bf7de8f7728ca1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.010754 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90327823332a4a63a4e9e16fcf74563f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.010965 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fbbca3bad5f4b448ae5afc4d5cfed14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.011519 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c36f0d063c9e498c96c83de1c4692745. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.012712 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 33d42f4c818340cda4ae615132fcf03f (DEFAULT_TABLE table=default.Table_9e22d18b516543cf9c179d97a1665544 [id=ae0591cb96f644ac827c33ed9c3c84e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.013039 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33d42f4c818340cda4ae615132fcf03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.013520 1627 tablet_service.cc:1505] Processing CreateTablet for tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 (DEFAULT_TABLE table=default.Table_9db3b592a2a64917b67ad65189a48a88 [id=b4b24248eaf6415488598c46f1cd2b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.014971 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 803) took 3422 ms. Trace:
I20251024 19:48:19.015301 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e927ba02569f42d2b6cd758f6e4d9113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.015991 1776 ts_tablet_manager.cc:1929] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:19.016120 1776 log.cc:1199] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/8a4cb86bc5e745d5b4ee932f0a38b3c7
I20251024 19:48:19.015928 1774 rpcz_store.cc:276] 1024 19:48:15.592969 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.593057 (+ 88us) service_pool.cc:225] Handling call
1024 19:48:19.014799 (+3421742us) ts_tablet_manager.cc:1151] Deleting tablet 63518b583ac94fa6abde420e176ec8ba
1024 19:48:19.014812 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.014948 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3421469}
I20251024 19:48:19.017169 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 1a332f4b383940a6a804c0aab94413d2 (DEFAULT_TABLE table=default.Table_ada6a6efca404633ba28f55fdcdcebec [id=c5cce7bfb00047cabd3ee47092550161]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.017760 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.018229 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 802) took 3494 ms. Trace:
I20251024 19:48:19.018548 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a332f4b383940a6a804c0aab94413d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.018517 1774 rpcz_store.cc:276] 1024 19:48:15.523228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.525597 (+ 2369us) service_pool.cc:225] Handling call
1024 19:48:19.017249 (+3491652us) ts_tablet_manager.cc:1151] Deleting tablet 2cbeaac19e834ecf98e5b8afc952b9da
1024 19:48:19.017259 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.017688 (+ 429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3340259}
I20251024 19:48:19.019500 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 804) took 3267 ms. Trace:
I20251024 19:48:19.019657 1776 ts_tablet_manager.cc:1950] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:19.020004 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 926) took 1739 ms. Trace:
I20251024 19:48:19.020200 1774 rpcz_store.cc:276] 1024 19:48:15.751803 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.752223 (+ 420us) service_pool.cc:225] Handling call
1024 19:48:19.019394 (+3267171us) ts_tablet_manager.cc:1151] Deleting tablet bf241a9566c34b7b911b69aaf4de06a2
1024 19:48:19.019403 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.019482 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3251531}
I20251024 19:48:19.021418 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 805) took 3228 ms. Trace:
I20251024 19:48:19.021816 1774 rpcz_store.cc:276] 1024 19:48:15.793273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.799590 (+ 6317us) service_pool.cc:225] Handling call
1024 19:48:19.020967 (+3221377us) ts_tablet_manager.cc:1151] Deleting tablet 605205170c994296a5ef0dc568788517
1024 19:48:19.020976 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.021400 (+ 424us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3221159}
I20251024 19:48:19.023193 1767 tablet_bootstrap.cc:654] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.024106 1621 tablet_service.cc:1505] Processing CreateTablet for tablet f1e414229de24e5ab96757e308253222 (DEFAULT_TABLE table=default.Table_83b888420618403f86e347bdb1c9f038 [id=170dd8adb74b4514a863501d847421a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.024528 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e414229de24e5ab96757e308253222. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.024893 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 818) took 2448 ms. Trace:
I20251024 19:48:19.020093 1776 rpcz_store.cc:276] 1024 19:48:17.280711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.281217 (+ 506us) service_pool.cc:225] Handling call
1024 19:48:18.921932 (+1640715us) ts_tablet_manager.cc:1151] Deleting tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7
1024 19:48:18.921946 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.930385 (+ 8439us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.015871 (+ 85486us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.019985 (+ 4114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1640559}
I20251024 19:48:19.029906 1776 tablet_replica.cc:333] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:19.030119 1776 raft_consensus.cc:2243] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:19.030481 1776 raft_consensus.cc:2272] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.031102 1776 ts_tablet_manager.cc:1916] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.032130 1586 leader_election.cc:304] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:19.032722 2920 raft_consensus.cc:2804] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:19.032824 2920 raft_consensus.cc:493] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:19.032732 3140 consensus_queue.cc:1048] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:19.032883 2920 raft_consensus.cc:3060] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.032848 3139 consensus_queue.cc:1048] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:19.041369 2923 raft_consensus.cc:515] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.061077 1775 ts_tablet_manager.cc:1929] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:19.061333 1775 log.cc:1199] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/3887fc21726846279fecb02672b9bc07
I20251024 19:48:19.061848 1775 ts_tablet_manager.cc:1950] T 3887fc21726846279fecb02672b9bc07 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:19.062860 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 838) took 4005 ms. Trace:
I20251024 19:48:19.063452 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 (table default.Table_5006c140e0964b569953c4f03495c697 [id=338beb9c6956484daef970b28d5e4b06]) successfully deleted
I20251024 19:48:19.070858 2923 leader_election.cc:290] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.047490 1372 raft_consensus.cc:3060] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.068852 2936 raft_consensus.cc:493] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.077737 2936 raft_consensus.cc:515] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.077868 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 9a94c856db4141cfb7ada7515d5b6e9e (DEFAULT_TABLE table=default.Table_8d2ad543f7a04e9e983cdaa997e94073 [id=2ebf9d4ca7114ee58d11cd9a745a09ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.078233 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a94c856db4141cfb7ada7515d5b6e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.078281 2920 raft_consensus.cc:515] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.078423 2936 leader_election.cc:290] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:19.078917 2920 leader_election.cc:290] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.085237 1774 rpcz_store.cc:276] 1024 19:48:16.576696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.576759 (+ 63us) service_pool.cc:225] Handling call
1024 19:48:19.024760 (+2448001us) ts_tablet_manager.cc:1151] Deleting tablet 265040336fb347cca1d52dd331a1418d
1024 19:48:19.024771 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.024871 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2447816}
I20251024 19:48:19.086516 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 819) took 2508 ms. Trace:
I20251024 19:48:19.089222 1774 rpcz_store.cc:276] 1024 19:48:16.578411 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.578456 (+ 45us) service_pool.cc:225] Handling call
1024 19:48:19.086361 (+2507905us) ts_tablet_manager.cc:1151] Deleting tablet 2e6ab4e8ffb84bf0961af481f0d4b180
1024 19:48:19.086372 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.086497 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2507761}
I20251024 19:48:19.090463 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 0a5034103c1e4e039df7fc9f0ab65d0c (DEFAULT_TABLE table=default.Table_effb50443d274effaa5039639c921715 [id=26bb760f582241fc8d6907a94d3deb44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.090422 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067631b128fd4a1e9ab72c9507dab6c4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:19.090778 1642 raft_consensus.cc:2393] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:19.090833 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5034103c1e4e039df7fc9f0ab65d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.093914 1612 tablet_service.cc:1505] Processing CreateTablet for tablet ae5f7f4ce4d3410da3f6c46dc40e7cff (DEFAULT_TABLE table=default.Table_68e3a1ca444149a2ad1ce7c1d54ad12b [id=53b9df9b421047ce8091942150364270]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.094282 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae5f7f4ce4d3410da3f6c46dc40e7cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.098713 3160 raft_consensus.cc:493] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10a9536207e343f188e0ef709081d04e)
I20251024 19:48:19.099826 3160 raft_consensus.cc:515] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.100782 3160 leader_election.cc:290] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:19.100772 1614 tablet_service.cc:1505] Processing CreateTablet for tablet f89fa648a6354f68a41552dc2c0604c1 (DEFAULT_TABLE table=default.Table_291336b400e94ac188244241309455d2 [id=19b791860c3646ceb9b08bc60aec9cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.101138 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f89fa648a6354f68a41552dc2c0604c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.101749 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 820) took 2466 ms. Trace:
I20251024 19:48:19.101891 1774 rpcz_store.cc:276] 1024 19:48:16.634908 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.635233 (+ 325us) service_pool.cc:225] Handling call
1024 19:48:19.101597 (+2466364us) ts_tablet_manager.cc:1151] Deleting tablet 9026d9e801294dd2bfdb9c90e45a0990
1024 19:48:19.101609 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.101730 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2466059}
I20251024 19:48:19.102444 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 821) took 2460 ms. Trace:
I20251024 19:48:19.102576 1774 rpcz_store.cc:276] 1024 19:48:16.642370 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.642436 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:19.102341 (+2459905us) ts_tablet_manager.cc:1151] Deleting tablet 41176885447e42409ee4363bb4d903f2
1024 19:48:19.102350 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.102428 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2459710}
I20251024 19:48:19.103021 1774 tablet_replica.cc:333] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:19.121657 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 121cfcb3a6dc4ed18484ac8eeabc13f7 (DEFAULT_TABLE table=default.Table_5783546824a846e7b084ac7e809885b9 [id=1f25b756b73d4fbf86a908f1a69866ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.121812 1629 tablet_service.cc:1505] Processing CreateTablet for tablet f0a0392d325f466dad6b4991b3c0cc45 (DEFAULT_TABLE table=default.Table_ea05abbf4e984c73a9e8df1fa0a505a0 [id=98d14b76a0c946e49259188a09852252]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.122192 1615 tablet_service.cc:1505] Processing CreateTablet for tablet d9367428428648f7ae07722ea09d70a7 (DEFAULT_TABLE table=default.Table_243a93945dd5416e94a881c0e85e5875 [id=11893740bece4b88b5fb6a651dc7bf86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.122265 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 121cfcb3a6dc4ed18484ac8eeabc13f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.125340 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 4438d3865f3c41d694428f31db6e4da7 (DEFAULT_TABLE table=default.Table_bf5080a43b0841818ccdbb98b65e05ba [id=a22c0f1885f94cdca5a77cba69b44f1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.125936 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9367428428648f7ae07722ea09d70a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.127625 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edb49884389c4dba957f0ee5af4ae0e4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:19.128222 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4438d3865f3c41d694428f31db6e4da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.128491 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0a0392d325f466dad6b4991b3c0cc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.129352 1768 tablet_bootstrap.cc:492] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:19.129591 1768 ts_tablet_manager.cc:1403] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.152s user 0.026s sys 0.027s
I20251024 19:48:19.130334 1768 raft_consensus.cc:359] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:19.130635 1768 raft_consensus.cc:385] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.130736 1768 raft_consensus.cc:740] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.131052 1768 consensus_queue.cc:260] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:19.131913 1768 ts_tablet_manager.cc:1434] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:19.132256 1768 tablet_bootstrap.cc:492] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:19.134122 1768 tablet_bootstrap.cc:654] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.135246 1696 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:19.137342 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 8aecec171ec548c48f2595b15860886e (DEFAULT_TABLE table=default.Table_a9bb007e715a4e80b9f3c77192d69c80 [id=170e25054d584dea9c79c9660bcbb256]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.137811 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aecec171ec548c48f2595b15860886e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.140080 1191 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 067631b128fd4a1e9ab72c9507dab6c4 (table default.Table_814a6dad46d54945b70e652b7e0f0032 [id=3f394c59d82b44928beb22f42312a4a3]). Expected version 1 got 0
I20251024 19:48:19.065342 2858 raft_consensus.cc:493] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.141752 2858 raft_consensus.cc:515] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.142402 2858 leader_election.cc:290] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.142542 1775 rpcz_store.cc:276] 1024 19:48:15.057166 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.057220 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:18.842493 (+3785273us) ts_tablet_manager.cc:1151] Deleting tablet 3887fc21726846279fecb02672b9bc07
1024 19:48:18.842503 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:18.953073 (+110570us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.060942 (+107869us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.062835 (+ 1893us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3785023}
I20251024 19:48:19.143005 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 839) took 3975 ms. Trace:
I20251024 19:48:19.143085 1775 rpcz_store.cc:276] 1024 19:48:15.167776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.167958 (+ 182us) service_pool.cc:225] Handling call
1024 19:48:19.142848 (+3974890us) ts_tablet_manager.cc:1151] Deleting tablet 95f62010a24f432c84c89345f9b02b2f
1024 19:48:19.142862 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.142988 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3974714}
I20251024 19:48:19.143410 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 840) took 3975 ms. Trace:
I20251024 19:48:19.143492 1775 rpcz_store.cc:276] 1024 19:48:15.167899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.167951 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:19.143298 (+3975347us) ts_tablet_manager.cc:1151] Deleting tablet 9f791083b4964ff9b0f5fc575478892c
1024 19:48:19.143307 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.143394 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3974892}
I20251024 19:48:19.143752 1775 tablet_replica.cc:333] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:19.143945 1775 raft_consensus.cc:2243] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:19.144106 1775 raft_consensus.cc:2272] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.144650 1775 ts_tablet_manager.cc:1916] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.079355 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067631b128fd4a1e9ab72c9507dab6c4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:19.145982 1505 raft_consensus.cc:2468] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:19.097787 1766 tablet_bootstrap.cc:492] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:19.146332 1766 ts_tablet_manager.cc:1403] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.194s user 0.006s sys 0.021s
I20251024 19:48:19.147249 1766 raft_consensus.cc:359] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.147416 1766 raft_consensus.cc:385] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.147472 1766 raft_consensus.cc:740] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.147670 1766 consensus_queue.cc:260] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.148041 1768 tablet_bootstrap.cc:492] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:19.148169 1768 ts_tablet_manager.cc:1403] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s
I20251024 19:48:19.148167 1766 ts_tablet_manager.cc:1434] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:19.148466 1766 tablet_bootstrap.cc:492] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:19.103338 1774 raft_consensus.cc:2243] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:19.148854 1774 pending_rounds.cc:70] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:19.148908 1774 pending_rounds.cc:77] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429373846736896 op_type: NO_OP noop_request { }
I20251024 19:48:19.149000 1774 raft_consensus.cc:2889] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:19.149082 1774 raft_consensus.cc:2272] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.149245 1768 raft_consensus.cc:359] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.149405 1768 raft_consensus.cc:385] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.149467 1768 raft_consensus.cc:740] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.149616 1774 ts_tablet_manager.cc:1916] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.149713 1768 consensus_queue.cc:260] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.150343 1768 ts_tablet_manager.cc:1434] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251024 19:48:19.150463 1766 tablet_bootstrap.cc:654] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.150631 1768 tablet_bootstrap.cc:492] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:19.150688 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9026d9e801294dd2bfdb9c90e45a0990" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:19.151163 1447 leader_election.cc:343] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:19.097990 1444 consensus_peers.cc:597] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d129bdc876e425aabcf1c99997e319a. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:19.151803 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 53848us
I20251024 19:48:19.124907 2857 raft_consensus.cc:493] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.152235 1308 leader_election.cc:304] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.152381 2857 raft_consensus.cc:515] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.152675 3160 raft_consensus.cc:2804] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:19.152714 1768 tablet_bootstrap.cc:654] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.152743 3160 raft_consensus.cc:493] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:19.152801 3160 raft_consensus.cc:3060] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.153044 2857 leader_election.cc:290] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.153995 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59956dad06064024ab171631ef6c1dca" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:19.154631 1447 leader_election.cc:343] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.155144 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edb49884389c4dba957f0ee5af4ae0e4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:19.155462 3160 raft_consensus.cc:515] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.155879 1308 leader_election.cc:304] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.156037 3160 leader_election.cc:290] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:19.156175 3160 raft_consensus.cc:2749] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:48:19.157115 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067631b128fd4a1e9ab72c9507dab6c4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:19.157348 1505 raft_consensus.cc:3060] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.160665 1505 raft_consensus.cc:2468] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:19.160714 1768 tablet_bootstrap.cc:492] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:19.160840 1768 ts_tablet_manager.cc:1403] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20251024 19:48:19.161239 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067631b128fd4a1e9ab72c9507dab6c4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:19.161509 1642 raft_consensus.cc:2393] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:19.161839 1768 raft_consensus.cc:359] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.162046 1768 raft_consensus.cc:385] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.162166 1768 raft_consensus.cc:740] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.162345 1308 leader_election.cc:304] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.162418 1768 consensus_queue.cc:260] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.162725 2936 raft_consensus.cc:2804] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:19.162829 2936 raft_consensus.cc:697] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:19.163094 1768 ts_tablet_manager.cc:1434] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:19.163033 2936 consensus_queue.cc:237] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.163336 1768 tablet_bootstrap.cc:492] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:19.165977 1768 tablet_bootstrap.cc:654] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.166517 1557 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
I20251024 19:48:19.167634 2936 raft_consensus.cc:493] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.167747 2936 raft_consensus.cc:515] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.168362 2936 leader_election.cc:290] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:19.169034 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a4cb86bc5e745d5b4ee932f0a38b3c7" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:19.169993 1305 leader_election.cc:343] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 8a4cb86bc5e745d5b4ee932f0a38b3c7
I20251024 19:48:19.170158 1305 leader_election.cc:304] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.170706 2936 raft_consensus.cc:2749] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:19.171257 1774 ts_tablet_manager.cc:1929] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:19.172425 1774 log.cc:1199] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/ce8c179e98944df0b510a0e8daf3920f
I20251024 19:48:19.172909 1774 ts_tablet_manager.cc:1950] T ce8c179e98944df0b510a0e8daf3920f P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:19.173298 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 825) took 2196 ms. Trace:
I20251024 19:48:19.174074 1775 ts_tablet_manager.cc:1929] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251024 19:48:19.174207 1774 rpcz_store.cc:276] 1024 19:48:16.976492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.976549 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:19.102978 (+2126429us) ts_tablet_manager.cc:1151] Deleting tablet ce8c179e98944df0b510a0e8daf3920f
1024 19:48:19.102986 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.152688 (+ 49702us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.171158 (+ 18470us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.173283 (+ 2125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2126236}
I20251024 19:48:19.174862 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 827) took 2164 ms. Trace:
I20251024 19:48:19.174986 1774 rpcz_store.cc:276] 1024 19:48:17.010557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.010613 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:19.174748 (+2164135us) ts_tablet_manager.cc:1151] Deleting tablet 3b8e8dbca79f4d72a2de51f105b209e4
1024 19:48:19.174758 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.174847 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2156413}
I20251024 19:48:19.175241 1775 log.cc:1199] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/8ba85164015448d4a688a473cce90c5c
I20251024 19:48:19.175832 1775 ts_tablet_manager.cc:1950] T 8ba85164015448d4a688a473cce90c5c P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:19.176092 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 831) took 2165 ms. Trace:
I20251024 19:48:19.176227 1774 rpcz_store.cc:276] 1024 19:48:17.010942 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.011004 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:19.175990 (+2164986us) ts_tablet_manager.cc:1151] Deleting tablet 4d0e4ffcdf184c5bafd99bf721b35951
1024 19:48:19.175999 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.176076 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2157586}
I20251024 19:48:19.176313 1191 catalog_manager.cc:5649] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43bec2f03fc849099e254f6335e9d532 (127.0.252.193). New cstate: current_term: 1 leader_uuid: "43bec2f03fc849099e254f6335e9d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: HEALTHY } } }
I20251024 19:48:19.176714 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 841) took 3905 ms. Trace:
I20251024 19:48:19.176788 1775 rpcz_store.cc:276] 1024 19:48:15.271659 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.272818 (+ 1159us) service_pool.cc:225] Handling call
1024 19:48:19.143707 (+3870889us) ts_tablet_manager.cc:1151] Deleting tablet 8ba85164015448d4a688a473cce90c5c
1024 19:48:19.143715 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.157829 (+ 14114us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.173971 (+ 16142us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.176697 (+ 2726us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3870685}
I20251024 19:48:19.177142 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 857) took 3656 ms. Trace:
I20251024 19:48:19.177209 1775 rpcz_store.cc:276] 1024 19:48:15.520419 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.520797 (+ 378us) service_pool.cc:225] Handling call
1024 19:48:19.177040 (+3656243us) ts_tablet_manager.cc:1151] Deleting tablet 584cbe0c396841f794e665aae3ebbd91
1024 19:48:19.177048 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.177129 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3656059}
I20251024 19:48:19.177492 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 858) took 3654 ms. Trace:
I20251024 19:48:19.176146 1768 tablet_bootstrap.cc:492] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:19.177699 1191 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 067631b128fd4a1e9ab72c9507dab6c4 (table default.Table_814a6dad46d54945b70e652b7e0f0032 [id=3f394c59d82b44928beb22f42312a4a3]). Expected version 1 got 0
I20251024 19:48:19.177866 1768 ts_tablet_manager.cc:1403] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20251024 19:48:19.178727 1768 raft_consensus.cc:359] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.178896 1768 raft_consensus.cc:385] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.178952 1768 raft_consensus.cc:740] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.179188 1768 consensus_queue.cc:260] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.179643 1768 ts_tablet_manager.cc:1434] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:19.183769 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 40f38dcc39b74cb19cde1cda7de875d8 (DEFAULT_TABLE table=default.Table_bcb7bf7c40d44f56afff3157570ddf05 [id=12cd4545270a488988958b82ed2372a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.184296 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40f38dcc39b74cb19cde1cda7de875d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.188244 1768 tablet_bootstrap.cc:492] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:19.190332 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 829) took 2179 ms. Trace:
I20251024 19:48:19.190469 1774 rpcz_store.cc:276] 1024 19:48:17.010775 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.011186 (+ 411us) service_pool.cc:225] Handling call
1024 19:48:19.179797 (+2168611us) ts_tablet_manager.cc:1151] Deleting tablet 0143a393711446fdbe6c798efa0c8936
1024 19:48:19.190136 (+ 10339us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.190303 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2161351}
I20251024 19:48:19.177558 1775 rpcz_store.cc:276] 1024 19:48:15.523357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.525787 (+ 2430us) service_pool.cc:225] Handling call
1024 19:48:19.177408 (+3651621us) ts_tablet_manager.cc:1151] Deleting tablet 2cbeaac19e834ecf98e5b8afc952b9da
1024 19:48:19.177415 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.177480 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3651461}
I20251024 19:48:19.194517 1768 tablet_bootstrap.cc:654] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.194717 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 859) took 3605 ms. Trace:
I20251024 19:48:19.194861 1775 rpcz_store.cc:276] 1024 19:48:15.589444 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.589522 (+ 78us) service_pool.cc:225] Handling call
1024 19:48:19.194500 (+3604978us) ts_tablet_manager.cc:1151] Deleting tablet 63518b583ac94fa6abde420e176ec8ba
1024 19:48:19.194510 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.194697 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3604737}
I20251024 19:48:19.195076 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 830) took 2184 ms. Trace:
I20251024 19:48:19.195175 1774 rpcz_store.cc:276] 1024 19:48:17.010863 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.011302 (+ 439us) service_pool.cc:225] Handling call
1024 19:48:19.190961 (+2179659us) ts_tablet_manager.cc:1151] Deleting tablet 4f00719c20934655bc744446194f7401
1024 19:48:19.194950 (+ 3989us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.195056 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2172464}
I20251024 19:48:19.195668 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 860) took 3588 ms. Trace:
I20251024 19:48:19.195823 1775 rpcz_store.cc:276] 1024 19:48:15.607626 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.607703 (+ 77us) service_pool.cc:225] Handling call
1024 19:48:19.195572 (+3587869us) ts_tablet_manager.cc:1151] Deleting tablet b79801594e95434cbb8fe7b75b14602e
1024 19:48:19.195582 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.195650 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3587603}
I20251024 19:48:19.196166 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 835) took 2151 ms. Trace:
I20251024 19:48:19.196277 1774 rpcz_store.cc:276] 1024 19:48:17.044363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.044860 (+ 497us) service_pool.cc:225] Handling call
1024 19:48:19.196045 (+2151185us) ts_tablet_manager.cc:1151] Deleting tablet ca6b8783039a4f5fbbda65c129805642
1024 19:48:19.196055 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.196145 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2151006}
I20251024 19:48:19.196411 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 861) took 3506 ms. Trace:
I20251024 19:48:19.197760 1774 tablet_replica.cc:333] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:19.197889 1775 rpcz_store.cc:276] 1024 19:48:15.689631 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.692437 (+ 2806us) service_pool.cc:225] Handling call
1024 19:48:19.196238 (+3503801us) ts_tablet_manager.cc:1151] Deleting tablet bf241a9566c34b7b911b69aaf4de06a2
1024 19:48:19.196249 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.196393 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3503468}
I20251024 19:48:19.198035 1774 raft_consensus.cc:2243] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:19.198444 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 862) took 3487 ms. Trace:
I20251024 19:48:19.198562 1775 rpcz_store.cc:276] 1024 19:48:15.711026 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:15.711210 (+ 184us) service_pool.cc:225] Handling call
1024 19:48:19.198349 (+3487139us) ts_tablet_manager.cc:1151] Deleting tablet 605205170c994296a5ef0dc568788517
1024 19:48:19.198357 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.198430 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3486892}
I20251024 19:48:19.199039 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 876) took 2621 ms. Trace:
I20251024 19:48:19.199156 1775 rpcz_store.cc:276] 1024 19:48:16.577794 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.578307 (+ 513us) service_pool.cc:225] Handling call
1024 19:48:19.198933 (+2620626us) ts_tablet_manager.cc:1151] Deleting tablet 265040336fb347cca1d52dd331a1418d
1024 19:48:19.198941 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.199026 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2620419}
I20251024 19:48:19.199314 1774 raft_consensus.cc:2272] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.199651 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 877) took 2620 ms. Trace:
I20251024 19:48:19.199775 1775 rpcz_store.cc:276] 1024 19:48:16.578772 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.578834 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:19.199533 (+2620699us) ts_tablet_manager.cc:1151] Deleting tablet 2e6ab4e8ffb84bf0961af481f0d4b180
1024 19:48:19.199542 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.199634 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2620560}
I20251024 19:48:19.199937 1774 ts_tablet_manager.cc:1916] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.203831 1775 tablet_replica.cc:333] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:19.204129 1775 raft_consensus.cc:2243] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:19.204370 1775 raft_consensus.cc:2272] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.205020 1775 ts_tablet_manager.cc:1916] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.205211 2936 raft_consensus.cc:493] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.205354 2936 raft_consensus.cc:515] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.206624 2936 leader_election.cc:290] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:19.206601 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f630fd412ca4c4f8570359c1678e6af" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:19.206965 1505 raft_consensus.cc:2393] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:19.207384 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f630fd412ca4c4f8570359c1678e6af" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:19.208026 1305 leader_election.cc:304] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.208544 2936 raft_consensus.cc:3060] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.209465 1774 ts_tablet_manager.cc:1929] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:19.209599 1774 log.cc:1199] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/607ae453221d422e9d07882830c02275
I20251024 19:48:19.209980 1774 ts_tablet_manager.cc:1950] T 607ae453221d422e9d07882830c02275 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:19.210323 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 840) took 2050 ms. Trace:
I20251024 19:48:19.210422 1774 rpcz_store.cc:276] 1024 19:48:17.159992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.160087 (+ 95us) service_pool.cc:225] Handling call
1024 19:48:19.197694 (+2037607us) ts_tablet_manager.cc:1151] Deleting tablet 607ae453221d422e9d07882830c02275
1024 19:48:19.197709 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.203823 (+ 6114us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.209353 (+ 5530us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.210303 (+ 950us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2037453}
I20251024 19:48:19.210763 1774 tablet_replica.cc:333] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:19.210958 1774 raft_consensus.cc:2243] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:19.211100 1774 raft_consensus.cc:2272] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.211457 2936 raft_consensus.cc:2749] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:19.211669 1774 ts_tablet_manager.cc:1916] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.218847 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 9489a01434a24d9ca94791252950725c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:19.222276 1768 tablet_bootstrap.cc:492] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:19.222430 1768 ts_tablet_manager.cc:1403] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.007s
I20251024 19:48:19.223161 1768 raft_consensus.cc:359] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:19.223326 1768 raft_consensus.cc:385] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.223374 1768 raft_consensus.cc:740] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.223572 1768 consensus_queue.cc:260] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:19.224027 1768 ts_tablet_manager.cc:1434] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:19.224774 1768 tablet_bootstrap.cc:492] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:19.224030 1774 ts_tablet_manager.cc:1929] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:19.229622 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 9b89a90bb4ca447fbaf34fe39001c261 (table default.Table_f37eb464f5144d1a8135a5f1a44cee4f [id=2fda7fc5a71646b79f5d848dca570135]) successfully deleted
I20251024 19:48:19.229887 1774 log.cc:1199] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/b1fa439afe5a48a79557019687319af6
W20251024 19:48:19.229893 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 520060a9adca4bb0b8616bcc006dce52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 520060a9adca4bb0b8616bcc006dce52 already in progress: creating tablet
I20251024 19:48:19.230144 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet b88a6a5237d146f5b6f082ca3af25b2b (table default.Table_6eb10ae51f3a43c2ab15f362708fba94 [id=f3080df28ee94876bb6b790cea45a4e8]) successfully deleted
I20251024 19:48:19.230333 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 701c93a600964c20a275f5942a2ff863 (table default.Table_7dc53b918ca7443b807133df217c0f75 [id=9ce6ac48924e45e3a482f71740edb4c0]) successfully deleted
I20251024 19:48:19.230508 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 4243f11bf2b34b30b10766c19e2c6409 (table default.Table_11634cbabbd6478484a0aed448ae9521 [id=d3b067d90c5943449b53edca7c6eb475]) successfully deleted
I20251024 19:48:19.230715 1774 ts_tablet_manager.cc:1950] T b1fa439afe5a48a79557019687319af6 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:19.230793 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet cef80e4d14bb4e719bceebf43041d079 (table default.Table_ade7993306ee438baebbf639e2109768 [id=6318d0634b77498a8c1ff8618b8ccf4b]) successfully deleted
W20251024 19:48:19.230980 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9f791083b4964ff9b0f5fc575478892c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f791083b4964ff9b0f5fc575478892c already in progress: creating tablet
W20251024 19:48:19.231153 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 95f62010a24f432c84c89345f9b02b2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95f62010a24f432c84c89345f9b02b2f already in progress: creating tablet
I20251024 19:48:19.231205 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 841) took 2070 ms. Trace:
I20251024 19:48:19.231330 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 8ba85164015448d4a688a473cce90c5c (table default.Table_fdf3a5f81257407d8b5d28c7f7cab1b7 [id=0fa321e2d4d847e69435f6de429d8779]) successfully deleted
W20251024 19:48:19.231539 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 584cbe0c396841f794e665aae3ebbd91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 584cbe0c396841f794e665aae3ebbd91 already in progress: creating tablet
I20251024 19:48:19.231384 1774 rpcz_store.cc:276] 1024 19:48:17.160499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.160542 (+ 43us) service_pool.cc:225] Handling call
1024 19:48:19.210714 (+2050172us) ts_tablet_manager.cc:1151] Deleting tablet b1fa439afe5a48a79557019687319af6
1024 19:48:19.210726 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.219536 (+ 8810us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.223941 (+ 4405us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.231188 (+ 7247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2050019}
W20251024 19:48:19.231720 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 63518b583ac94fa6abde420e176ec8ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63518b583ac94fa6abde420e176ec8ba already in progress: creating tablet
W20251024 19:48:19.231881 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2cbeaac19e834ecf98e5b8afc952b9da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cbeaac19e834ecf98e5b8afc952b9da already in progress: creating tablet
W20251024 19:48:19.232048 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet bf241a9566c34b7b911b69aaf4de06a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf241a9566c34b7b911b69aaf4de06a2 already in progress: creating tablet
W20251024 19:48:19.232221 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 605205170c994296a5ef0dc568788517 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 605205170c994296a5ef0dc568788517 already in progress: creating tablet
I20251024 19:48:19.232268 1774 tablet_replica.cc:333] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
W20251024 19:48:19.232398 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 265040336fb347cca1d52dd331a1418d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 265040336fb347cca1d52dd331a1418d already in progress: creating tablet
W20251024 19:48:19.232599 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2e6ab4e8ffb84bf0961af481f0d4b180 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e6ab4e8ffb84bf0961af481f0d4b180 already in progress: creating tablet
I20251024 19:48:19.232740 1774 raft_consensus.cc:2243] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:19.232784 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9026d9e801294dd2bfdb9c90e45a0990 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9026d9e801294dd2bfdb9c90e45a0990 already in progress: creating tablet
I20251024 19:48:19.233011 1774 raft_consensus.cc:2272] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.234181 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9026d9e801294dd2bfdb9c90e45a0990" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:19.234584 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59956dad06064024ab171631ef6c1dca" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:19.235745 1444 leader_election.cc:343] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.235878 1444 leader_election.cc:304] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:19.236054 1444 raft_consensus.cc:2686] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251024 19:48:19.236337 1444 leader_election.cc:343] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.236429 1444 leader_election.cc:304] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.236786 2857 raft_consensus.cc:2749] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:19.244264 1766 tablet_bootstrap.cc:492] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:19.244649 1766 ts_tablet_manager.cc:1403] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.007s
I20251024 19:48:19.245204 1774 ts_tablet_manager.cc:1916] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.247014 1768 tablet_bootstrap.cc:654] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:19.252633 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 41176885447e42409ee4363bb4d903f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41176885447e42409ee4363bb4d903f2 already in progress: creating tablet
I20251024 19:48:19.252912 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet ce8c179e98944df0b510a0e8daf3920f (table default.Table_dedd7e29ac324761b962fbd47c7c25e9 [id=870c629198ed4f53b9317b1340e75ea2]) successfully deleted
W20251024 19:48:19.253124 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3b8e8dbca79f4d72a2de51f105b209e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b8e8dbca79f4d72a2de51f105b209e4 already in progress: creating tablet
W20251024 19:48:19.253324 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4d0e4ffcdf184c5bafd99bf721b35951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d0e4ffcdf184c5bafd99bf721b35951 already in progress: creating tablet
W20251024 19:48:19.253520 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 0143a393711446fdbe6c798efa0c8936 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0143a393711446fdbe6c798efa0c8936 already in progress: creating tablet
W20251024 19:48:19.253762 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4f00719c20934655bc744446194f7401 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f00719c20934655bc744446194f7401 already in progress: creating tablet
W20251024 19:48:19.253947 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet ca6b8783039a4f5fbbda65c129805642 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca6b8783039a4f5fbbda65c129805642 already in progress: creating tablet
I20251024 19:48:19.254092 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 607ae453221d422e9d07882830c02275 (table default.Table_e962c1d3850943a0a9e86f5882d1c518 [id=2656b4a933364e6a98f1ee5f91f0b2ba]) successfully deleted
I20251024 19:48:19.254266 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet b1fa439afe5a48a79557019687319af6 (table default.Table_231e4d38da7a4fe4b4e6b33e854aa1d5 [id=77778be7b00c4eccbce4716fc4a864f7]) successfully deleted
I20251024 19:48:19.254530 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 16bc737104ab46b3b071a0760dbc847d (table default.Table_775e17f58fef4f2b850035a03e411a08 [id=6239ceae506442b79bbe79c48813b49e]) successfully deleted
I20251024 19:48:19.254750 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 3887fc21726846279fecb02672b9bc07 (table default.Table_e378095ddecc4dcf801a1568160c34b4 [id=450b69bd10cf4c629cc3f8dadc790312]) successfully deleted
W20251024 19:48:19.254896 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 95f62010a24f432c84c89345f9b02b2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95f62010a24f432c84c89345f9b02b2f already in progress: creating tablet
W20251024 19:48:19.255043 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9f791083b4964ff9b0f5fc575478892c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f791083b4964ff9b0f5fc575478892c already in progress: creating tablet
I20251024 19:48:19.255169 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 8ba85164015448d4a688a473cce90c5c (table default.Table_fdf3a5f81257407d8b5d28c7f7cab1b7 [id=0fa321e2d4d847e69435f6de429d8779]) successfully deleted
W20251024 19:48:19.255304 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 584cbe0c396841f794e665aae3ebbd91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 584cbe0c396841f794e665aae3ebbd91 already in progress: creating tablet
W20251024 19:48:19.255440 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2cbeaac19e834ecf98e5b8afc952b9da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cbeaac19e834ecf98e5b8afc952b9da already in progress: creating tablet
W20251024 19:48:19.255581 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 63518b583ac94fa6abde420e176ec8ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63518b583ac94fa6abde420e176ec8ba already in progress: creating tablet
W20251024 19:48:19.255750 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b79801594e95434cbb8fe7b75b14602e because the tablet was not found. No further retry: Not found: Tablet not found: b79801594e95434cbb8fe7b75b14602e
W20251024 19:48:19.255921 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet bf241a9566c34b7b911b69aaf4de06a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf241a9566c34b7b911b69aaf4de06a2 already in progress: creating tablet
W20251024 19:48:19.256064 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 605205170c994296a5ef0dc568788517 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 605205170c994296a5ef0dc568788517 already in progress: creating tablet
W20251024 19:48:19.256214 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 265040336fb347cca1d52dd331a1418d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 265040336fb347cca1d52dd331a1418d already in progress: creating tablet
W20251024 19:48:19.256345 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2e6ab4e8ffb84bf0961af481f0d4b180 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e6ab4e8ffb84bf0961af481f0d4b180 already in progress: creating tablet
I20251024 19:48:19.257004 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 3324755a5a184a609fae5f7a25d58e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:19.269613 1766 raft_consensus.cc:359] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.272987 1766 raft_consensus.cc:385] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.273110 1766 raft_consensus.cc:740] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.273394 1766 consensus_queue.cc:260] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.274000 1766 ts_tablet_manager.cc:1434] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20251024 19:48:19.277256 1766 tablet_bootstrap.cc:492] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:19.280741 1776 ts_tablet_manager.cc:1929] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:19.280951 1776 log.cc:1199] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/639836c656f44a819036f4d00e055bd9
I20251024 19:48:19.287653 1768 tablet_bootstrap.cc:492] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:19.287837 1768 ts_tablet_manager.cc:1403] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.009s
I20251024 19:48:19.289134 1768 raft_consensus.cc:359] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.289304 1768 raft_consensus.cc:385] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.289373 1768 raft_consensus.cc:740] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.289664 1768 consensus_queue.cc:260] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.290998 1768 ts_tablet_manager.cc:1434] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20251024 19:48:19.293366 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 260657us
I20251024 19:48:19.296432 1776 ts_tablet_manager.cc:1950] T 639836c656f44a819036f4d00e055bd9 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:19.296870 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 921) took 2016 ms. Trace:
I20251024 19:48:19.297302 1370 raft_consensus.cc:3060] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.299052 1768 tablet_bootstrap.cc:492] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:19.306911 2936 raft_consensus.cc:493] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.307137 2936 raft_consensus.cc:515] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.307789 2936 leader_election.cc:290] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:19.308969 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f89a3bbeb9b4d57a49078729110d951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:19.309993 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 346cb0bb8a0d43f7a15b788a303a6bcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:19.310637 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c82f60819e7a442bbc0cc04659e14e87" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:19.310847 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 346cb0bb8a0d43f7a15b788a303a6bcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:19.311476 3039 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 5610871893708698182) tablet 00000000000000000000000000000000 cur holder 0x617000041700
I20251024 19:48:19.311666 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet af918843c74940888c709d9a05aa6e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:19.312669 1774 ts_tablet_manager.cc:1929] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:19.312988 1774 log.cc:1199] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/7c77f6bbf7b145389ca6cf74ddd3aa3f
I20251024 19:48:19.313480 1774 ts_tablet_manager.cc:1950] T 7c77f6bbf7b145389ca6cf74ddd3aa3f P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:19.316407 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 842) took 2155 ms. Trace:
I20251024 19:48:19.317639 1774 rpcz_store.cc:276] 1024 19:48:17.160943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.160987 (+ 44us) service_pool.cc:225] Handling call
1024 19:48:19.232201 (+2071214us) ts_tablet_manager.cc:1151] Deleting tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f
1024 19:48:19.232216 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.265548 (+ 33332us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.292087 (+ 26539us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.316371 (+ 24284us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2071071}
I20251024 19:48:19.318413 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet af918843c74940888c709d9a05aa6e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:18 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:19.319628 1305 leader_election.cc:343] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: c82f60819e7a442bbc0cc04659e14e87
I20251024 19:48:19.319823 1305 leader_election.cc:304] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.320732 2936 raft_consensus.cc:2749] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:19.320473 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 843) took 2146 ms. Trace:
I20251024 19:48:19.297042 1776 rpcz_store.cc:276] 1024 19:48:17.280163 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.281404 (+ 1241us) service_pool.cc:225] Handling call
1024 19:48:19.029849 (+1748445us) ts_tablet_manager.cc:1151] Deleting tablet 639836c656f44a819036f4d00e055bd9
1024 19:48:19.029861 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.221079 (+191218us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.280628 (+ 59549us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.296852 (+ 16224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1748325}
I20251024 19:48:19.322670 1345 tablet_service.cc:1505] Processing CreateTablet for tablet c36f0d063c9e498c96c83de1c4692745 (DEFAULT_TABLE table=default.Table_266203ab5d8d40979ab181d5715a4348 [id=733f3fdc70e44f4c838d04bf66b08c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.322909 1776 tablet_replica.cc:333] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:19.323181 1776 raft_consensus.cc:2243] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:19.323217 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c36f0d063c9e498c96c83de1c4692745. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.323611 1776 raft_consensus.cc:2272] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.323959 1768 tablet_bootstrap.cc:654] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.324501 1776 ts_tablet_manager.cc:1916] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.321017 1774 rpcz_store.cc:276] 1024 19:48:17.174114 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.174167 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:19.320230 (+2146063us) ts_tablet_manager.cc:1151] Deleting tablet b8d30b2fdeec410fa582a05db0968ebb
1024 19:48:19.320252 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.320435 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2145876}
I20251024 19:48:19.328202 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 639836c656f44a819036f4d00e055bd9 (table default.Table_6eb10ae51f3a43c2ab15f362708fba94 [id=f3080df28ee94876bb6b790cea45a4e8]) successfully deleted
I20251024 19:48:19.328796 1774 tablet_replica.cc:333] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:19.329185 1774 raft_consensus.cc:2243] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:19.329516 1774 pending_rounds.cc:70] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:19.329597 1774 pending_rounds.cc:77] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429377511059456 op_type: NO_OP noop_request { }
I20251024 19:48:19.329699 1774 raft_consensus.cc:2889] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:19.329794 1774 raft_consensus.cc:2272] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.330468 1774 ts_tablet_manager.cc:1916] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.340065 1365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067631b128fd4a1e9ab72c9507dab6c4" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:19.323087 1766 tablet_bootstrap.cc:654] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.341924 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0405765c6146cbbb187af96a60be2e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:19.342326 1511 raft_consensus.cc:3060] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.345427 1511 raft_consensus.cc:2468] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:19.330567 1506 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "067631b128fd4a1e9ab72c9507dab6c4" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:19.345657 1475 tablet_service.cc:1505] Processing CreateTablet for tablet c36f0d063c9e498c96c83de1c4692745 (DEFAULT_TABLE table=default.Table_266203ab5d8d40979ab181d5715a4348 [id=733f3fdc70e44f4c838d04bf66b08c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.346037 1506 raft_consensus.cc:2393] T 067631b128fd4a1e9ab72c9507dab6c4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 43bec2f03fc849099e254f6335e9d532 in this term.
I20251024 19:48:19.346199 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c36f0d063c9e498c96c83de1c4692745. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.353765 1365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0405765c6146cbbb187af96a60be2e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:19.355060 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 1a7468b6f68046fa91ce076ffe3f97a5 (DEFAULT_TABLE table=default.Table_4d0d05989b3844cd8813c527af4862a1 [id=98b4201eaf8a4d059f67fb980cdced3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.355525 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a7468b6f68046fa91ce076ffe3f97a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.357004 2857 raft_consensus.cc:493] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.357161 2857 raft_consensus.cc:515] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.357833 2857 leader_election.cc:290] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.358335 1365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10891ff5bf0d4b93bcab9faa5f7346f1" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:19.358465 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10891ff5bf0d4b93bcab9faa5f7346f1" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:19.358866 1447 leader_election.cc:343] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:19.359000 1444 leader_election.cc:343] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.359074 1447 leader_election.cc:304] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.359442 2857 raft_consensus.cc:2749] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:19.359804 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf4596b333244cc9e9e35a128721e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:19.360237 1767 tablet_bootstrap.cc:492] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:19.362399 2857 raft_consensus.cc:493] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.362563 2857 raft_consensus.cc:515] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.363716 2857 leader_election.cc:290] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.366272 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 1a7468b6f68046fa91ce076ffe3f97a5 (DEFAULT_TABLE table=default.Table_4d0d05989b3844cd8813c527af4862a1 [id=98b4201eaf8a4d059f67fb980cdced3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.364256 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3d80885fcc642b8a75da30cd4abecbd" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:19.366884 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a7468b6f68046fa91ce076ffe3f97a5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:19.367453 1444 leader_election.cc:343] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.362490 1767 ts_tablet_manager.cc:1403] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.431s user 0.021s sys 0.012s
W20251024 19:48:19.371532 1586 leader_election.cc:343] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.373441 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3d80885fcc642b8a75da30cd4abecbd" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:19.374176 1447 leader_election.cc:343] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.374300 1447 leader_election.cc:304] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.374760 2857 raft_consensus.cc:2749] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:19.378732 1473 tablet_service.cc:1505] Processing CreateTablet for tablet cb1f850b3e104707a53d540ad5752d4d (DEFAULT_TABLE table=default.Table_12fe00934de64b41af4f5fa5980dd476 [id=ea3808eabb2a430a9313732c4a1d2c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.379159 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb1f850b3e104707a53d540ad5752d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.377489 3175 tablet.cc:1717] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532: Skipping requested alter to schema version 2, tablet already version 2
I20251024 19:48:19.381742 1774 ts_tablet_manager.cc:1929] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:19.383097 1774 log.cc:1199] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/f51c56337ebb450aaf62dcc81ef63ae8
I20251024 19:48:19.383577 1774 ts_tablet_manager.cc:1950] T f51c56337ebb450aaf62dcc81ef63ae8 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:19.389412 1344 tablet_service.cc:1505] Processing CreateTablet for tablet cb1f850b3e104707a53d540ad5752d4d (DEFAULT_TABLE table=default.Table_12fe00934de64b41af4f5fa5980dd476 [id=ea3808eabb2a430a9313732c4a1d2c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.376354 1767 raft_consensus.cc:359] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.390101 1767 raft_consensus.cc:385] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.390178 1767 raft_consensus.cc:740] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.391531 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 844) took 2030 ms. Trace:
I20251024 19:48:19.391626 1774 rpcz_store.cc:276] 1024 19:48:17.360934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.360995 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:19.328736 (+1967741us) ts_tablet_manager.cc:1151] Deleting tablet f51c56337ebb450aaf62dcc81ef63ae8
1024 19:48:19.328748 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.354374 (+ 25626us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.381500 (+ 27126us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.391512 (+ 10012us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28544,"tablet-delete.queue_time_us":1967548}
I20251024 19:48:19.391969 1774 tablet_replica.cc:333] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:19.392010 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb1f850b3e104707a53d540ad5752d4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.392151 1774 raft_consensus.cc:2243] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:19.392293 1774 raft_consensus.cc:2272] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.392830 1774 ts_tablet_manager.cc:1916] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.394167 1767 consensus_queue.cc:260] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.394706 1767 ts_tablet_manager.cc:1434] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20251024 19:48:19.398360 1768 tablet_bootstrap.cc:492] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:19.398639 1768 ts_tablet_manager.cc:1403] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.100s user 0.011s sys 0.027s
I20251024 19:48:19.404320 1775 ts_tablet_manager.cc:1929] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:19.404549 1775 log.cc:1199] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/9026d9e801294dd2bfdb9c90e45a0990
I20251024 19:48:19.404963 1775 ts_tablet_manager.cc:1950] T 9026d9e801294dd2bfdb9c90e45a0990 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:19.405838 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 878) took 2770 ms. Trace:
I20251024 19:48:19.406001 1775 rpcz_store.cc:276] 1024 19:48:16.635871 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.635933 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:19.203778 (+2567845us) ts_tablet_manager.cc:1151] Deleting tablet 9026d9e801294dd2bfdb9c90e45a0990
1024 19:48:19.203789 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.213883 (+ 10094us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.404192 (+190309us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.405819 (+ 1627us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2567629}
I20251024 19:48:19.406586 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 879) took 2763 ms. Trace:
W20251024 19:48:19.407702 1586 consensus_peers.cc:597] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16bc737104ab46b3b071a0760dbc847d. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:19.409391 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 1a332f4b383940a6a804c0aab94413d2 (DEFAULT_TABLE table=default.Table_ada6a6efca404633ba28f55fdcdcebec [id=c5cce7bfb00047cabd3ee47092550161]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.409555 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 1a332f4b383940a6a804c0aab94413d2 (DEFAULT_TABLE table=default.Table_ada6a6efca404633ba28f55fdcdcebec [id=c5cce7bfb00047cabd3ee47092550161]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.409972 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a332f4b383940a6a804c0aab94413d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.411973 1768 raft_consensus.cc:359] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.412142 1768 raft_consensus.cc:385] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.412194 1768 raft_consensus.cc:740] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.412168 1586 leader_election.cc:304] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:19.412400 1768 consensus_queue.cc:260] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.412736 3141 raft_consensus.cc:2749] T 067631b128fd4a1e9ab72c9507dab6c4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251024 19:48:19.412917 1768 ts_tablet_manager.cc:1434] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251024 19:48:19.409627 1767 tablet_bootstrap.cc:492] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:19.413800 1586 leader_election.cc:304] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:19.414778 1774 ts_tablet_manager.cc:1929] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:19.414887 1774 log.cc:1199] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/505e4fb0755a4caaa462e5f5154bfee2
I20251024 19:48:19.415261 1774 ts_tablet_manager.cc:1950] T 505e4fb0755a4caaa462e5f5154bfee2 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:19.414280 1776 ts_tablet_manager.cc:1929] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:19.415470 1776 log.cc:1199] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/4193750331874d7ba9bbe6279c3a8e37
I20251024 19:48:19.415796 1776 ts_tablet_manager.cc:1950] T 4193750331874d7ba9bbe6279c3a8e37 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:19.416262 3163 raft_consensus.cc:2804] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:19.416368 3163 raft_consensus.cc:697] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:19.416673 3163 consensus_queue.cc:237] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.418668 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a332f4b383940a6a804c0aab94413d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.418747 1768 tablet_bootstrap.cc:492] T d6de2c21796f4bc4904f88b654c3baec P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:19.420800 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 27729f3779454bf9bf7de8f7728ca1a6 (DEFAULT_TABLE table=default.Table_23a248d91f5d45fbbf4bf104efd8bfb2 [id=1863f145669846098f7568da3262dd3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.421221 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27729f3779454bf9bf7de8f7728ca1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.421741 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 845) took 2057 ms. Trace:
I20251024 19:48:19.421820 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 27729f3779454bf9bf7de8f7728ca1a6 (DEFAULT_TABLE table=default.Table_23a248d91f5d45fbbf4bf104efd8bfb2 [id=1863f145669846098f7568da3262dd3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.421849 1774 rpcz_store.cc:276] 1024 19:48:17.364277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.364332 (+ 55us) service_pool.cc:225] Handling call
1024 19:48:19.391922 (+2027590us) ts_tablet_manager.cc:1151] Deleting tablet 505e4fb0755a4caaa462e5f5154bfee2
1024 19:48:19.391933 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.397267 (+ 5334us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.414684 (+ 17417us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.421719 (+ 7035us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2027359}
I20251024 19:48:19.422248 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27729f3779454bf9bf7de8f7728ca1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.422333 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 846) took 2052 ms. Trace:
I20251024 19:48:19.422408 1774 rpcz_store.cc:276] 1024 19:48:17.369735 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.369789 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:19.422192 (+2052403us) ts_tablet_manager.cc:1151] Deleting tablet eb6b7df6176e4bccacf7a9059cee5477
1024 19:48:19.422204 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.422317 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2052196}
I20251024 19:48:19.423431 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 847) took 2020 ms. Trace:
I20251024 19:48:19.423753 1774 rpcz_store.cc:276] 1024 19:48:17.402966 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.403020 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:19.423310 (+2020290us) ts_tablet_manager.cc:1151] Deleting tablet 40338a7939be4240b44dc54cc77ef020
1024 19:48:19.423321 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.423415 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2020101}
I20251024 19:48:19.424108 1774 tablet_replica.cc:333] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:19.424350 3137 consensus_queue.cc:1048] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:19.424440 1774 raft_consensus.cc:2243] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:19.424643 1774 raft_consensus.cc:2272] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:19.424822 1586 consensus_peers.cc:597] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be8126459ef541f39bc133f27c7d0f59. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:19.414954 1767 tablet_bootstrap.cc:654] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.425221 1774 ts_tablet_manager.cc:1916] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.430753 1768 tablet_bootstrap.cc:654] T d6de2c21796f4bc4904f88b654c3baec P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.430866 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 81a3de69a13c459b9b7c02761ab7261e (table default.Table_b5a66c4e9cd8408782690b66cdb87473 [id=3d82ad2f9e184043aed4ec0b05df724f]). Expected version 1 got 0
I20251024 19:48:19.431488 1366 raft_consensus.cc:3060] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:19.436496 1444 consensus_peers.cc:597] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c82f60819e7a442bbc0cc04659e14e87. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:19.438375 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 927) took 2144 ms. Trace:
I20251024 19:48:19.438827 1341 tablet_service.cc:1505] Processing CreateTablet for tablet e927ba02569f42d2b6cd758f6e4d9113 (DEFAULT_TABLE table=default.Table_669783bf95a24c658906c5ea84b932b1 [id=231c6fbfbc4840428aaebf311b8f051e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.439214 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e927ba02569f42d2b6cd758f6e4d9113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.439219 1483 tablet_service.cc:1505] Processing CreateTablet for tablet e927ba02569f42d2b6cd758f6e4d9113 (DEFAULT_TABLE table=default.Table_669783bf95a24c658906c5ea84b932b1 [id=231c6fbfbc4840428aaebf311b8f051e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.439595 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e927ba02569f42d2b6cd758f6e4d9113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.442173 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 90327823332a4a63a4e9e16fcf74563f (DEFAULT_TABLE table=default.Table_9423eab92cc74554b6a926d3c438fd30 [id=23639a33342f4a2396dc37d8905efc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.442508 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90327823332a4a63a4e9e16fcf74563f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.442404 3141 consensus_queue.cc:1048] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:19.443621 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 90327823332a4a63a4e9e16fcf74563f (DEFAULT_TABLE table=default.Table_9423eab92cc74554b6a926d3c438fd30 [id=23639a33342f4a2396dc37d8905efc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.443964 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90327823332a4a63a4e9e16fcf74563f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.443933 1776 rpcz_store.cc:276] 1024 19:48:17.293668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.293742 (+ 74us) service_pool.cc:225] Handling call
1024 19:48:19.322791 (+2029049us) ts_tablet_manager.cc:1151] Deleting tablet 4193750331874d7ba9bbe6279c3a8e37
1024 19:48:19.322801 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.362202 (+ 39401us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.414195 (+ 51993us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.438342 (+ 24147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10813,"tablet-delete.queue_time_us":2028811}
I20251024 19:48:19.444450 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 4193750331874d7ba9bbe6279c3a8e37 (table default.Table_18cdd8018c304deba24dd3429fdb48fc [id=3172f9d08e8649c289a02ad1cd61177d]) successfully deleted
I20251024 19:48:19.446317 1776 tablet_replica.cc:333] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:19.446753 3141 consensus_queue.cc:1048] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:19.449704 1776 raft_consensus.cc:2243] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:19.452733 1776 raft_consensus.cc:2272] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.453361 3141 consensus_queue.cc:1048] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:19.464396 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 97f6223d9e8b4f80b70b7515108d8492 (DEFAULT_TABLE table=default.Table_6cfafb2b827d4dba9de0de5bb5472304 [id=253d20af9be945b6a55bad0baf545bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.406728 1775 rpcz_store.cc:276] 1024 19:48:16.642809 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.642879 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:19.406473 (+2763594us) ts_tablet_manager.cc:1151] Deleting tablet 41176885447e42409ee4363bb4d903f2
1024 19:48:19.406482 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.406571 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2763424}
I20251024 19:48:19.464847 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97f6223d9e8b4f80b70b7515108d8492. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.465134 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 886) took 2489 ms. Trace:
I20251024 19:48:19.465286 1775 rpcz_store.cc:276] 1024 19:48:16.975417 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.975688 (+ 271us) service_pool.cc:225] Handling call
1024 19:48:19.465017 (+2489329us) ts_tablet_manager.cc:1151] Deleting tablet 9b89a90bb4ca447fbaf34fe39001c261
1024 19:48:19.465027 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.465115 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2489093}
I20251024 19:48:19.465730 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 97f6223d9e8b4f80b70b7515108d8492 (DEFAULT_TABLE table=default.Table_6cfafb2b827d4dba9de0de5bb5472304 [id=253d20af9be945b6a55bad0baf545bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.466184 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97f6223d9e8b4f80b70b7515108d8492. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.454895 1767 tablet_bootstrap.cc:492] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:19.471621 1767 ts_tablet_manager.cc:1403] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.000s
I20251024 19:48:19.465847 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 887) took 2490 ms. Trace:
I20251024 19:48:19.472070 1775 rpcz_store.cc:276] 1024 19:48:16.975499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.975959 (+ 460us) service_pool.cc:225] Handling call
1024 19:48:19.465740 (+2489781us) ts_tablet_manager.cc:1151] Deleting tablet 758df87acbdb4adaa83e1ecbc44e7cf1
1024 19:48:19.465751 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.465833 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2489603}
I20251024 19:48:19.472729 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 888) took 2497 ms. Trace:
I20251024 19:48:19.472872 1775 rpcz_store.cc:276] 1024 19:48:16.975608 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.976166 (+ 558us) service_pool.cc:225] Handling call
1024 19:48:19.472603 (+2496437us) ts_tablet_manager.cc:1151] Deleting tablet fdfa5eea3b674a3389a156d1c72204df
1024 19:48:19.472613 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.472711 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2496293}
I20251024 19:48:19.473435 1775 tablet_replica.cc:333] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:19.474373 1766 tablet_bootstrap.cc:492] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:19.474884 1766 ts_tablet_manager.cc:1403] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.198s user 0.004s sys 0.002s
I20251024 19:48:19.475689 1776 ts_tablet_manager.cc:1916] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.476709 1766 raft_consensus.cc:359] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.477085 1766 raft_consensus.cc:385] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.477283 1766 raft_consensus.cc:740] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.477612 1767 raft_consensus.cc:359] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.477880 1767 raft_consensus.cc:385] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.477967 1767 raft_consensus.cc:740] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.477710 1766 consensus_queue.cc:260] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.478253 1767 consensus_queue.cc:260] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.478921 1767 ts_tablet_manager.cc:1434] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251024 19:48:19.478941 1766 ts_tablet_manager.cc:1434] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251024 19:48:19.479326 1767 tablet_bootstrap.cc:492] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:19.479705 1766 tablet_bootstrap.cc:492] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:19.481897 1775 raft_consensus.cc:2243] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:19.482297 1775 raft_consensus.cc:2272] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.483212 1775 ts_tablet_manager.cc:1916] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.484865 1766 tablet_bootstrap.cc:654] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.488567 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 33d42f4c818340cda4ae615132fcf03f (DEFAULT_TABLE table=default.Table_9e22d18b516543cf9c179d97a1665544 [id=ae0591cb96f644ac827c33ed9c3c84e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.488720 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 (DEFAULT_TABLE table=default.Table_9db3b592a2a64917b67ad65189a48a88 [id=b4b24248eaf6415488598c46f1cd2b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.489084 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33d42f4c818340cda4ae615132fcf03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.495405 1766 tablet_bootstrap.cc:492] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:19.495569 1766 ts_tablet_manager.cc:1403] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20251024 19:48:19.496340 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 1fbbca3bad5f4b448ae5afc4d5cfed14 (DEFAULT_TABLE table=default.Table_33762464873048acb73f8d26e167301e [id=59b4d3aff42249eb8ff8f4292f690c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.496428 1766 raft_consensus.cc:359] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.496620 1766 raft_consensus.cc:385] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.496711 1766 raft_consensus.cc:740] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.496949 1766 consensus_queue.cc:260] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.497255 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fbbca3bad5f4b448ae5afc4d5cfed14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.497494 1766 ts_tablet_manager.cc:1434] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:19.497767 1766 tablet_bootstrap.cc:492] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:19.500236 1766 tablet_bootstrap.cc:654] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.502074 2936 raft_consensus.cc:493] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.502213 2936 raft_consensus.cc:515] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.502301 1477 tablet_service.cc:1505] Processing CreateTablet for tablet f1e414229de24e5ab96757e308253222 (DEFAULT_TABLE table=default.Table_83b888420618403f86e347bdb1c9f038 [id=170dd8adb74b4514a863501d847421a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.502692 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e414229de24e5ab96757e308253222. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.503219 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.518959 1766 tablet_bootstrap.cc:492] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:19.519207 1766 ts_tablet_manager.cc:1403] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.002s
I20251024 19:48:19.520480 1766 raft_consensus.cc:359] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.520751 1766 raft_consensus.cc:385] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.520844 1766 raft_consensus.cc:740] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.521075 1766 consensus_queue.cc:260] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.521723 1766 ts_tablet_manager.cc:1434] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:19.529050 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 33d42f4c818340cda4ae615132fcf03f (DEFAULT_TABLE table=default.Table_9e22d18b516543cf9c179d97a1665544 [id=ae0591cb96f644ac827c33ed9c3c84e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.530239 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62667092d7e244cca1020ec220ea9124" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:19.530499 1504 raft_consensus.cc:2393] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:19.532747 1192 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet d6de2c21796f4bc4904f88b654c3baec (table default.Table_1a079b981b694617b3baf9897dfb37ae [id=18ec057e920e4c21ae8cbe298058b6f3]). Expected version 2 got 0
I20251024 19:48:19.534215 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 9a94c856db4141cfb7ada7515d5b6e9e (DEFAULT_TABLE table=default.Table_8d2ad543f7a04e9e983cdaa997e94073 [id=2ebf9d4ca7114ee58d11cd9a745a09ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.534734 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a94c856db4141cfb7ada7515d5b6e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.536296 3137 raft_consensus.cc:493] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.536466 3137 raft_consensus.cc:515] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.537185 3137 leader_election.cc:290] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.537530 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129debb55f4a4b7295a8ad88e55ebc83" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:19.537748 1504 raft_consensus.cc:2468] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:19.538017 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 129debb55f4a4b7295a8ad88e55ebc83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:19.538897 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129debb55f4a4b7295a8ad88e55ebc83" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:19.539458 1586 leader_election.cc:304] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:19.539800 3137 raft_consensus.cc:2804] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:19.539886 3137 raft_consensus.cc:493] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:19.539944 3137 raft_consensus.cc:3060] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:19.539920 1586 leader_election.cc:343] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.543571 3137 raft_consensus.cc:515] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.544550 3137 leader_election.cc:290] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.544634 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 0a5034103c1e4e039df7fc9f0ab65d0c (DEFAULT_TABLE table=default.Table_effb50443d274effaa5039639c921715 [id=26bb760f582241fc8d6907a94d3deb44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.544770 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 40f38dcc39b74cb19cde1cda7de875d8 (DEFAULT_TABLE table=default.Table_bcb7bf7c40d44f56afff3157570ddf05 [id=12cd4545270a488988958b82ed2372a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.545035 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5034103c1e4e039df7fc9f0ab65d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.545181 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 121cfcb3a6dc4ed18484ac8eeabc13f7 (DEFAULT_TABLE table=default.Table_5783546824a846e7b084ac7e809885b9 [id=1f25b756b73d4fbf86a908f1a69866ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.545552 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 121cfcb3a6dc4ed18484ac8eeabc13f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.545886 1487 tablet_service.cc:1505] Processing CreateTablet for tablet ae5f7f4ce4d3410da3f6c46dc40e7cff (DEFAULT_TABLE table=default.Table_68e3a1ca444149a2ad1ce7c1d54ad12b [id=53b9df9b421047ce8091942150364270]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.546217 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae5f7f4ce4d3410da3f6c46dc40e7cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.549780 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40f38dcc39b74cb19cde1cda7de875d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.554139 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129debb55f4a4b7295a8ad88e55ebc83" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:19.554322 1504 raft_consensus.cc:3060] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.557119 1504 raft_consensus.cc:2468] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:19.558809 1766 tablet_bootstrap.cc:492] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:19.560683 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129debb55f4a4b7295a8ad88e55ebc83" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:19.560858 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 (DEFAULT_TABLE table=default.Table_9db3b592a2a64917b67ad65189a48a88 [id=b4b24248eaf6415488598c46f1cd2b8a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:19.561724 1586 leader_election.cc:343] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.561941 1586 leader_election.cc:304] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:19.562604 1335 tablet_service.cc:1505] Processing CreateTablet for tablet f1e414229de24e5ab96757e308253222 (DEFAULT_TABLE table=default.Table_83b888420618403f86e347bdb1c9f038 [id=170dd8adb74b4514a863501d847421a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.562798 1766 tablet_bootstrap.cc:654] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.562894 1338 tablet_service.cc:1505] Processing CreateTablet for tablet f89fa648a6354f68a41552dc2c0604c1 (DEFAULT_TABLE table=default.Table_291336b400e94ac188244241309455d2 [id=19b791860c3646ceb9b08bc60aec9cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.562976 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33d42f4c818340cda4ae615132fcf03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.562574 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 9a94c856db4141cfb7ada7515d5b6e9e (DEFAULT_TABLE table=default.Table_8d2ad543f7a04e9e983cdaa997e94073 [id=2ebf9d4ca7114ee58d11cd9a745a09ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.563233 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a94c856db4141cfb7ada7515d5b6e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.564232 2920 raft_consensus.cc:2804] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:19.564366 2920 raft_consensus.cc:697] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:19.564616 2920 consensus_queue.cc:237] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.565277 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 1fbbca3bad5f4b448ae5afc4d5cfed14 (DEFAULT_TABLE table=default.Table_33762464873048acb73f8d26e167301e [id=59b4d3aff42249eb8ff8f4292f690c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.565634 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fbbca3bad5f4b448ae5afc4d5cfed14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.565977 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f89fa648a6354f68a41552dc2c0604c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.566401 1477 tablet_service.cc:1505] Processing CreateTablet for tablet f89fa648a6354f68a41552dc2c0604c1 (DEFAULT_TABLE table=default.Table_291336b400e94ac188244241309455d2 [id=19b791860c3646ceb9b08bc60aec9cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.566771 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f89fa648a6354f68a41552dc2c0604c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.567627 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e414229de24e5ab96757e308253222. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.568010 2936 leader_election.cc:290] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:19.569855 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.571702 1350 tablet_service.cc:1505] Processing CreateTablet for tablet ae5f7f4ce4d3410da3f6c46dc40e7cff (DEFAULT_TABLE table=default.Table_68e3a1ca444149a2ad1ce7c1d54ad12b [id=53b9df9b421047ce8091942150364270]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.572052 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae5f7f4ce4d3410da3f6c46dc40e7cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.572563 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62667092d7e244cca1020ec220ea9124" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:19.573190 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 4438d3865f3c41d694428f31db6e4da7 (DEFAULT_TABLE table=default.Table_bf5080a43b0841818ccdbb98b65e05ba [id=a22c0f1885f94cdca5a77cba69b44f1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.573333 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 0a5034103c1e4e039df7fc9f0ab65d0c (DEFAULT_TABLE table=default.Table_effb50443d274effaa5039639c921715 [id=26bb760f582241fc8d6907a94d3deb44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.573724 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4438d3865f3c41d694428f31db6e4da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.576759 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 8aecec171ec548c48f2595b15860886e (DEFAULT_TABLE table=default.Table_a9bb007e715a4e80b9f3c77192d69c80 [id=170e25054d584dea9c79c9660bcbb256]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.577114 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aecec171ec548c48f2595b15860886e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.577232 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 40f38dcc39b74cb19cde1cda7de875d8 (DEFAULT_TABLE table=default.Table_bcb7bf7c40d44f56afff3157570ddf05 [id=12cd4545270a488988958b82ed2372a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.579807 1766 tablet_bootstrap.cc:492] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:19.580066 1766 ts_tablet_manager.cc:1403] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20251024 19:48:19.581328 1766 raft_consensus.cc:359] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.581899 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 121cfcb3a6dc4ed18484ac8eeabc13f7 (DEFAULT_TABLE table=default.Table_5783546824a846e7b084ac7e809885b9 [id=1f25b756b73d4fbf86a908f1a69866ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.582278 1766 raft_consensus.cc:385] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.582486 1766 raft_consensus.cc:740] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.582653 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5034103c1e4e039df7fc9f0ab65d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.582849 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 121cfcb3a6dc4ed18484ac8eeabc13f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.582870 1305 leader_election.cc:304] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.583097 2936 raft_consensus.cc:3060] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.583550 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40f38dcc39b74cb19cde1cda7de875d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.583671 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 8aecec171ec548c48f2595b15860886e (DEFAULT_TABLE table=default.Table_a9bb007e715a4e80b9f3c77192d69c80 [id=170e25054d584dea9c79c9660bcbb256]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.583982 1766 consensus_queue.cc:260] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.585036 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 4438d3865f3c41d694428f31db6e4da7 (DEFAULT_TABLE table=default.Table_bf5080a43b0841818ccdbb98b65e05ba [id=a22c0f1885f94cdca5a77cba69b44f1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.585553 1342 tablet_service.cc:1505] Processing CreateTablet for tablet f0a0392d325f466dad6b4991b3c0cc45 (DEFAULT_TABLE table=default.Table_ea05abbf4e984c73a9e8df1fa0a505a0 [id=98d14b76a0c946e49259188a09852252]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.585901 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aecec171ec548c48f2595b15860886e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.589672 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0a0392d325f466dad6b4991b3c0cc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.590409 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4438d3865f3c41d694428f31db6e4da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.596287 3160 raft_consensus.cc:493] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.596455 3160 raft_consensus.cc:515] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:19.597036 3160 leader_election.cc:290] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:19.597007 1352 tablet_service.cc:1505] Processing CreateTablet for tablet d9367428428648f7ae07722ea09d70a7 (DEFAULT_TABLE table=default.Table_243a93945dd5416e94a881c0e85e5875 [id=11893740bece4b88b5fb6a651dc7bf86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.597339 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9367428428648f7ae07722ea09d70a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.597039 1488 tablet_service.cc:1505] Processing CreateTablet for tablet f0a0392d325f466dad6b4991b3c0cc45 (DEFAULT_TABLE table=default.Table_ea05abbf4e984c73a9e8df1fa0a505a0 [id=98d14b76a0c946e49259188a09852252]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.598315 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0a0392d325f466dad6b4991b3c0cc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.597590 1485 tablet_service.cc:1505] Processing CreateTablet for tablet d9367428428648f7ae07722ea09d70a7 (DEFAULT_TABLE table=default.Table_243a93945dd5416e94a881c0e85e5875 [id=11893740bece4b88b5fb6a651dc7bf86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:19.598214 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639836c656f44a819036f4d00e055bd9" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:19.599488 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9367428428648f7ae07722ea09d70a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:19.607414 2857 raft_consensus.cc:493] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.607568 2857 raft_consensus.cc:515] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.608114 1766 ts_tablet_manager.cc:1434] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20251024 19:48:19.608356 2857 leader_election.cc:290] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.609000 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f965111ac60841dbb9e6ecc21ff3fec0" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:19.609516 1444 leader_election.cc:343] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.609894 1766 tablet_bootstrap.cc:492] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:19.611624 1766 tablet_bootstrap.cc:654] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.611676 1340 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:19.613081 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f965111ac60841dbb9e6ecc21ff3fec0" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:19.613754 1447 leader_election.cc:343] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.613899 1447 leader_election.cc:304] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.614450 2857 raft_consensus.cc:2749] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:19.614988 1305 leader_election.cc:343] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 639836c656f44a819036f4d00e055bd9
I20251024 19:48:19.615211 1305 leader_election.cc:304] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.615644 3160 raft_consensus.cc:2749] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:19.620016 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet e38ef2ac19d14f9c91fa8abe86cc7987 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:19.620344 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c89ac6d4e142431a9d712dc0f1e32987"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:19.620940 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c89ac6d4e142431a9d712dc0f1e32987 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:19.622944 2857 raft_consensus.cc:493] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.623109 2857 raft_consensus.cc:515] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.623881 2857 leader_election.cc:290] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.624614 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2a68bb1dc6c483388784b9bb19ce065" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:19.620036 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8713def8ab7543788d6466f5d28125a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:19.625231 1444 leader_election.cc:343] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:19.625540 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8713def8ab7543788d6466f5d28125a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:19.625929 1767 tablet_bootstrap.cc:700] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97: Time spent opening tablet: real 0.107s user 0.001s sys 0.000s
I20251024 19:48:19.626228 1767 tablet_bootstrap.cc:654] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.629806 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet e38ef2ac19d14f9c91fa8abe86cc7987
I20251024 19:48:19.629997 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2a68bb1dc6c483388784b9bb19ce065" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:19.630260 1192 catalog_manager.cc:6719] ReplaceTablet: tablet e38ef2ac19d14f9c91fa8abe86cc7987 deleted and replaced by tablet 6b1a3523954448de93357510e6c1ec4e
I20251024 19:48:19.630941 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet e38ef2ac19d14f9c91fa8abe86cc7987 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1a3523954448de93357510e6c1ec4e) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:19.631326 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet b240965605904533bd2af99f61aaef26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:19.631484 1447 leader_election.cc:343] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.631676 1447 leader_election.cc:304] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.632164 2857 raft_consensus.cc:2749] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:19.632637 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet c82f60819e7a442bbc0cc04659e14e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:00 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:19.632822 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f630fd412ca4c4f8570359c1678e6af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:19.633129 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet 639836c656f44a819036f4d00e055bd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:19.634959 1776 ts_tablet_manager.cc:1929] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:19.635066 1776 log.cc:1199] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/f83b9290f00145a780318a43249eec12
I20251024 19:48:19.635414 1776 ts_tablet_manager.cc:1950] T f83b9290f00145a780318a43249eec12 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:19.636180 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 928) took 2341 ms. Trace:
I20251024 19:48:19.636261 1776 rpcz_store.cc:276] 1024 19:48:17.294766 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.294855 (+ 89us) service_pool.cc:225] Handling call
1024 19:48:19.446241 (+2151386us) ts_tablet_manager.cc:1151] Deleting tablet f83b9290f00145a780318a43249eec12
1024 19:48:19.446258 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.519725 (+ 73467us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.634850 (+115125us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.636160 (+ 1310us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14356,"tablet-delete.queue_time_us":2151220}
I20251024 19:48:19.637385 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet f83b9290f00145a780318a43249eec12 (table default.Table_420ebc050cf440e9a59a18e26f7b83af [id=be1cc9d047554ea6a9afa531fb08a81d]) successfully deleted
I20251024 19:48:19.655582 1776 tablet_replica.cc:333] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:19.657634 1776 raft_consensus.cc:2243] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:19.658676 1776 raft_consensus.cc:2272] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.657027 1767 tablet_bootstrap.cc:492] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:19.659827 1767 ts_tablet_manager.cc:1403] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.181s user 0.005s sys 0.000s
I20251024 19:48:19.660801 1767 raft_consensus.cc:359] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.661132 1767 raft_consensus.cc:385] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.661257 1767 raft_consensus.cc:740] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.661631 1767 consensus_queue.cc:260] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.662334 1767 ts_tablet_manager.cc:1434] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:19.666746 3163 consensus_queue.cc:1048] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:19.666954 1767 tablet_bootstrap.cc:492] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:19.668238 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 6424b2347ef642feb38a9596f2d2be1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:19.668716 1767 tablet_bootstrap.cc:654] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.675803 1364 raft_consensus.cc:3060] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.677551 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c9d965a31a7f4ba3bed33813b02d7dfb" } new_table_name: "default.Table_6b2935707be843948a0ebf57ce97b78c" modify_external_catalogs: true
I20251024 19:48:19.686914 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d10102cfa0fa4eb9b806bacdd5066807" } new_table_name: "default.Table_5c558f9e1e6c4f06ad6bc5f3e11af622" modify_external_catalogs: true
I20251024 19:48:19.688985 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3e8db9d6df6542088fef2a3b687daddd" } modify_external_catalogs: true
I20251024 19:48:19.689105 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b71e2a66a1cf4e47bf6cffbbef51f618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:19.689210 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3e8db9d6df6542088fef2a3b687daddd" } modify_external_catalogs: true
W20251024 19:48:19.689708 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b71e2a66a1cf4e47bf6cffbbef51f618 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:19.689687 1776 ts_tablet_manager.cc:1916] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.695016 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_673109e8b8e744f99120d9f081b1dd9d" } modify_external_catalogs: true
I20251024 19:48:19.695327 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_673109e8b8e744f99120d9f081b1dd9d" } modify_external_catalogs: true
I20251024 19:48:19.696972 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet 3fd8aa1d9aca429dbfe46f472ebbf53b from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:19.698014 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_90159463cd7741cba6ccb29461b104cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:19.697371 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_be3f9c7f025449e6bef4abd44dc74fd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:19.698709 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90159463cd7741cba6ccb29461b104cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:19.698868 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be3f9c7f025449e6bef4abd44dc74fd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:19.700794 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_692697818af34700ad79bbc08b8deb43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:19.701311 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_692697818af34700ad79bbc08b8deb43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:19.704669 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8d2ad543f7a04e9e983cdaa997e94073" } modify_external_catalogs: true
I20251024 19:48:19.704855 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8d2ad543f7a04e9e983cdaa997e94073" } modify_external_catalogs: true
I20251024 19:48:19.706521 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet e38ef2ac19d14f9c91fa8abe86cc7987 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1a3523954448de93357510e6c1ec4e) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:19.710305 2857 raft_consensus.cc:493] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.710513 2857 raft_consensus.cc:515] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.711292 2857 leader_election.cc:290] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.711654 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1edc55caab904722bf19a1c71df601cf" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:19.712571 1444 leader_election.cc:343] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.714385 3137 consensus_queue.cc:1048] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:19.720458 3160 raft_consensus.cc:493] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.720695 3160 raft_consensus.cc:515] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:19.721556 3160 leader_election.cc:290] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:19.722563 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b240965605904533bd2af99f61aaef26" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:19.724567 1776 ts_tablet_manager.cc:1929] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:19.727420 1776 log.cc:1199] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/9489a01434a24d9ca94791252950725c
I20251024 19:48:19.728969 1776 ts_tablet_manager.cc:1950] T 9489a01434a24d9ca94791252950725c P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:19.731446 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 931) took 2220 ms. Trace:
I20251024 19:48:19.734160 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1f888bb5a760477c88e82d58b11aa364" } new_table_name: "default.Table_5fd8256a4e4949a883edb79127167a15" modify_external_catalogs: true
W20251024 19:48:19.735496 1179 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d610b007a0054067ab8a1278580a98d6 (table default.Table_6a4162ff35fb4f1285ec130d29c15320 [id=ac2c4408a8b14a5cb1252f5c8f811000], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet d610b007a0054067ab8a1278580a98d6 consensus state has no leader
I20251024 19:48:19.735992 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 9026d9e801294dd2bfdb9c90e45a0990 (table default.Table_9e14814a2db64e0b9602e41f926f3b62 [id=1841054cd03e4989a236ebf134fa7755]) successfully deleted
W20251024 19:48:19.736542 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 41176885447e42409ee4363bb4d903f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41176885447e42409ee4363bb4d903f2 already in progress: creating tablet
W20251024 19:48:19.736799 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9b89a90bb4ca447fbaf34fe39001c261 because the tablet was not found. No further retry: Not found: Tablet not found: 9b89a90bb4ca447fbaf34fe39001c261
W20251024 19:48:19.737154 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 758df87acbdb4adaa83e1ecbc44e7cf1 because the tablet was not found. No further retry: Not found: Tablet not found: 758df87acbdb4adaa83e1ecbc44e7cf1
W20251024 19:48:19.737187 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e65583d0ea84a3ea10ae11462eac663 consensus state has no leader
W20251024 19:48:19.737731 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet fdfa5eea3b674a3389a156d1c72204df because the tablet was not found. No further retry: Not found: Tablet not found: fdfa5eea3b674a3389a156d1c72204df
I20251024 19:48:19.737672 1776 rpcz_store.cc:276] 1024 19:48:17.511184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.511233 (+ 49us) service_pool.cc:225] Handling call
1024 19:48:19.655503 (+2144270us) ts_tablet_manager.cc:1151] Deleting tablet 9489a01434a24d9ca94791252950725c
1024 19:48:19.655519 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.709087 (+ 53568us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.724447 (+ 15360us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.731419 (+ 6972us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2144095}
I20251024 19:48:19.736270 1767 tablet_bootstrap.cc:492] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:19.738390 1767 ts_tablet_manager.cc:1403] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.008s
I20251024 19:48:19.738411 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 9489a01434a24d9ca94791252950725c (table default.Table_1c89d244042a4852a9cb4113f6726332 [id=ba207dd9990d4a0e85eaefbb4908dd02]) successfully deleted
W20251024 19:48:19.738924 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e602f14c45104517b7618b163519a499 consensus state has no leader
I20251024 19:48:19.739777 1767 raft_consensus.cc:359] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.743824 1767 raft_consensus.cc:385] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.744199 1767 raft_consensus.cc:740] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.744781 1767 consensus_queue.cc:260] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.739006 1776 tablet_replica.cc:333] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:19.746462 1776 raft_consensus.cc:2243] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:19.749325 1775 ts_tablet_manager.cc:1929] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:19.749776 1776 raft_consensus.cc:2272] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.749994 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9ba52b13507546c4b00c9109363aacc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:19.750625 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ba52b13507546c4b00c9109363aacc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:19.750952 1776 ts_tablet_manager.cc:1916] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.755299 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4834ef486a9b47a9976759474b0b2ca1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:19.756036 1767 ts_tablet_manager.cc:1434] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
W20251024 19:48:19.756136 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4834ef486a9b47a9976759474b0b2ca1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:19.769176 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 7c77f6bbf7b145389ca6cf74ddd3aa3f (table default.Table_a1d09325af844d71b290d95b8a4aaf12 [id=b62ec7bcad7c41b1b946e9ebf2f4bf39]) successfully deleted
W20251024 19:48:19.769455 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b8d30b2fdeec410fa582a05db0968ebb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8d30b2fdeec410fa582a05db0968ebb already in progress: creating tablet
I20251024 19:48:19.769784 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet f51c56337ebb450aaf62dcc81ef63ae8 (table default.Table_efcd2af1913c4a73ba5148272aa880aa [id=01a8150e819b4075810a95720eaa5840]) successfully deleted
I20251024 19:48:19.770102 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 505e4fb0755a4caaa462e5f5154bfee2 (table default.Table_74e4f24f12db403686548924c83c5191 [id=508d6cdc5d4a494bad31eaf36383e214]) successfully deleted
W20251024 19:48:19.770359 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet eb6b7df6176e4bccacf7a9059cee5477 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb6b7df6176e4bccacf7a9059cee5477 already in progress: creating tablet
W20251024 19:48:19.770586 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 40338a7939be4240b44dc54cc77ef020 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40338a7939be4240b44dc54cc77ef020 already in progress: creating tablet
I20251024 19:48:19.766072 1766 tablet_bootstrap.cc:492] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:19.772537 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet d840888183ec4ef7bb245363c5bac322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:19.772964 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 59956dad06064024ab171631ef6c1dca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:19.777407 1766 ts_tablet_manager.cc:1403] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.168s user 0.039s sys 0.074s
I20251024 19:48:19.778098 1766 raft_consensus.cc:359] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.778249 1766 raft_consensus.cc:385] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.778290 1766 raft_consensus.cc:740] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.778471 1766 consensus_queue.cc:260] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.778923 1766 ts_tablet_manager.cc:1434] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 19:48:19.779155 1766 tablet_bootstrap.cc:492] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:19.780570 1766 tablet_bootstrap.cc:654] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:19.782011 1586 consensus_peers.cc:597] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:19.782224 2920 raft_consensus.cc:493] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.782336 2920 raft_consensus.cc:515] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.782936 2920 leader_election.cc:290] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.783789 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf4596b333244cc9e9e35a128721e4a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:19.783963 1504 raft_consensus.cc:2468] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:19.784759 2857 raft_consensus.cc:493] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.784873 2857 raft_consensus.cc:515] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.785430 2857 leader_election.cc:290] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.786113 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28005301f9de48e5826f46c79aeb5fe5" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:19.786609 1444 leader_election.cc:343] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.786901 1586 leader_election.cc:304] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:19.787184 2920 raft_consensus.cc:2804] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:19.787248 2920 raft_consensus.cc:493] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:19.787299 2920 raft_consensus.cc:3060] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.792775 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet e38ef2ac19d14f9c91fa8abe86cc7987 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1a3523954448de93357510e6c1ec4e) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:19.793991 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf4596b333244cc9e9e35a128721e4a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:19.794502 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b240965605904533bd2af99f61aaef26" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:19.796438 1586 leader_election.cc:343] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.796669 1775 log.cc:1199] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/c82f60819e7a442bbc0cc04659e14e87
I20251024 19:48:19.797331 1775 ts_tablet_manager.cc:1950] T c82f60819e7a442bbc0cc04659e14e87 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:19.797384 2920 raft_consensus.cc:515] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.798902 2920 leader_election.cc:290] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.801438 1767 tablet_bootstrap.cc:492] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:19.802184 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4bb5f923e4074b398f16e0f53223fbf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:19.802798 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bb5f923e4074b398f16e0f53223fbf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:19.805030 1767 tablet_bootstrap.cc:654] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.806180 1776 ts_tablet_manager.cc:1929] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:19.808983 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_630eb735037f4337989ff580a3a6eee4" } new_table_name: "default.Table_2134712eb3c54d96879eef43e8fb1d2a" modify_external_catalogs: true
I20251024 19:48:19.808954 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 50cea450eda9485daba40bf875a1b90c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:19.791167 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1edc55caab904722bf19a1c71df601cf" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:19.798436 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf4596b333244cc9e9e35a128721e4a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:19.816896 1504 raft_consensus.cc:3060] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.818329 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c3cc8727161245ab88c9c67c4b45546f" } new_table_name: "default.Table_43953a13cfc64df2920d9de139b89be0" modify_external_catalogs: true
I20251024 19:48:19.818778 1768 tablet_bootstrap.cc:492] T d6de2c21796f4bc4904f88b654c3baec P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:19.818912 1768 ts_tablet_manager.cc:1403] T d6de2c21796f4bc4904f88b654c3baec P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.400s user 0.008s sys 0.008s
I20251024 19:48:19.819679 1768 raft_consensus.cc:359] T d6de2c21796f4bc4904f88b654c3baec P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:19.819845 1768 raft_consensus.cc:385] T d6de2c21796f4bc4904f88b654c3baec P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.819900 1768 raft_consensus.cc:740] T d6de2c21796f4bc4904f88b654c3baec P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.820088 1768 consensus_queue.cc:260] T d6de2c21796f4bc4904f88b654c3baec P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:19.820307 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9a94c856db4141cfb7ada7515d5b6e9e
I20251024 19:48:19.821511 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a94c856db4141cfb7ada7515d5b6e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:19.821516 1447 leader_election.cc:343] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.822265 1447 leader_election.cc:304] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.822391 1504 raft_consensus.cc:2468] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:19.822897 1586 leader_election.cc:304] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:19.824436 1774 ts_tablet_manager.cc:1929] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:19.824471 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf4596b333244cc9e9e35a128721e4a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:19.824541 1774 log.cc:1199] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/e29abfa3bc8944aea7d1acd03e3eb897
I20251024 19:48:19.824446 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2b311ba111854dabaeaaff96e18ee699"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:19.824797 3163 raft_consensus.cc:2804] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:19.824904 3163 raft_consensus.cc:697] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
W20251024 19:48:19.825026 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b311ba111854dabaeaaff96e18ee699 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:19.825114 3163 consensus_queue.cc:237] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.825958 2857 raft_consensus.cc:2749] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:19.826354 1586 leader_election.cc:343] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.829200 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a94c856db4141cfb7ada7515d5b6e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:19.834246 1767 tablet_bootstrap.cc:492] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:19.843354 2936 raft_consensus.cc:2749] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:19.844013 1305 leader_election.cc:304] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.845947 2936 raft_consensus.cc:2749] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:19.847333 1774 ts_tablet_manager.cc:1950] T e29abfa3bc8944aea7d1acd03e3eb897 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:19.847681 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 848) took 2427 ms. Trace:
I20251024 19:48:19.847827 1774 rpcz_store.cc:276] 1024 19:48:17.420599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.420651 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:19.424067 (+2003416us) ts_tablet_manager.cc:1151] Deleting tablet e29abfa3bc8944aea7d1acd03e3eb897
1024 19:48:19.424075 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.584910 (+160835us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.824327 (+239417us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.847667 (+ 23340us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2003209}
I20251024 19:48:19.848814 1774 tablet_replica.cc:333] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:19.849184 1774 raft_consensus.cc:2243] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:19.849476 1774 raft_consensus.cc:2272] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.850256 1774 ts_tablet_manager.cc:1916] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.851449 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a94c856db4141cfb7ada7515d5b6e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:19.840731 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28005301f9de48e5826f46c79aeb5fe5" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:19.857283 1447 leader_election.cc:343] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.857534 1447 leader_election.cc:304] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:19.858526 2857 raft_consensus.cc:2749] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:19.861016 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet e29abfa3bc8944aea7d1acd03e3eb897 (table default.Table_55339e02660843c3a8536a4b7a06831e [id=ba30503223dd4979824ec9c1a50a8ae2]) successfully deleted
I20251024 19:48:19.862246 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 884) took 2887 ms. Trace:
W20251024 19:48:19.861686 1370 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.500s user 0.032s sys 0.047s
I20251024 19:48:19.838462 2920 consensus_queue.cc:1048] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:19.815418 1776 log.cc:1199] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/3324755a5a184a609fae5f7a25d58e4c
I20251024 19:48:19.865597 1776 ts_tablet_manager.cc:1950] T 3324755a5a184a609fae5f7a25d58e4c P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:19.866638 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 930) took 2355 ms. Trace:
I20251024 19:48:19.866788 1776 rpcz_store.cc:276] 1024 19:48:17.511038 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.511354 (+ 316us) service_pool.cc:225] Handling call
1024 19:48:19.738948 (+2227594us) ts_tablet_manager.cc:1151] Deleting tablet 3324755a5a184a609fae5f7a25d58e4c
1024 19:48:19.738960 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.755043 (+ 16083us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.806058 (+ 51015us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.866616 (+ 60558us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2842,"tablet-delete.queue_time_us":2227422}
I20251024 19:48:19.867357 1776 tablet_replica.cc:333] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:19.867332 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 3324755a5a184a609fae5f7a25d58e4c (table default.Table_2f543df6b1194a7195ecd656134e4b17 [id=e908b8ad13614c3c90e0f8aaafd0421f]) successfully deleted
I20251024 19:48:19.867790 1776 raft_consensus.cc:2243] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:19.868184 1776 raft_consensus.cc:2272] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.868868 1767 ts_tablet_manager.cc:1403] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.068s user 0.007s sys 0.000s
I20251024 19:48:19.870245 1767 raft_consensus.cc:359] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.870573 1775 rpcz_store.cc:276] 1024 19:48:16.975256 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.977630 (+ 2374us) service_pool.cc:225] Handling call
1024 19:48:19.473379 (+2495749us) ts_tablet_manager.cc:1151] Deleting tablet c82f60819e7a442bbc0cc04659e14e87
1024 19:48:19.473387 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.640844 (+167457us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.749213 (+108369us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.862227 (+113014us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4,"tablet-delete.queue_time_us":2495524}
I20251024 19:48:19.862619 1766 tablet_bootstrap.cc:492] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:19.871893 1766 ts_tablet_manager.cc:1403] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.093s user 0.005s sys 0.004s
I20251024 19:48:19.872597 1766 raft_consensus.cc:359] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.872823 1766 raft_consensus.cc:385] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.872908 1766 raft_consensus.cc:740] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.873152 1766 consensus_queue.cc:260] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:19.873783 1766 ts_tablet_manager.cc:1434] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251024 19:48:19.874109 1766 tablet_bootstrap.cc:492] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:19.874305 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet c82f60819e7a442bbc0cc04659e14e87 (table default.Table_6f376570019346199a3c8c77d879fa1c [id=c5f7c2469f394bba8b91e2f20170963e]) successfully deleted
I20251024 19:48:19.875566 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet 3a712f52cd114e9bb07c2717e5d728fd from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:19.876278 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 885) took 2900 ms. Trace:
I20251024 19:48:19.876361 1775 rpcz_store.cc:276] 1024 19:48:16.975342 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.978277 (+ 2935us) service_pool.cc:225] Handling call
1024 19:48:19.876172 (+2897895us) ts_tablet_manager.cc:1151] Deleting tablet 7352d84226db467dad31ae5941842b80
1024 19:48:19.876182 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.876262 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2897571}
W20251024 19:48:19.876896 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7352d84226db467dad31ae5941842b80 because the tablet was not found. No further retry: Not found: Tablet not found: 7352d84226db467dad31ae5941842b80
W20251024 19:48:19.877130 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe9cb79e470948e5b0af65b6b668913a consensus state has no leader
I20251024 19:48:19.877341 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 3ef6d244c498452ca79cd6c6d5c003df from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:19.909718 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b2992a738b440d798f5e61cc45ec088 consensus state has no leader
I20251024 19:48:19.910522 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_33fc19a822a142afb669502687f7980c" } modify_external_catalogs: true
I20251024 19:48:19.910712 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_33fc19a822a142afb669502687f7980c" } modify_external_catalogs: true
I20251024 19:48:19.872732 1776 ts_tablet_manager.cc:1916] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:19.886724 2923 consensus_queue.cc:1048] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:19.913277 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet 9287f340fb554904986f6c18b7356701 from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:19.914033 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85c7134bb2f046319868d41df161572c consensus state has no leader
I20251024 19:48:19.914505 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_97ae72e3d285443a8435906795593d52" } modify_external_catalogs: true
I20251024 19:48:19.914664 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_97ae72e3d285443a8435906795593d52" } modify_external_catalogs: true
I20251024 19:48:19.919062 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 62667092d7e244cca1020ec220ea9124 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:19.919461 1767 raft_consensus.cc:385] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:19.922259 1767 raft_consensus.cc:740] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:19.929284 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f6df45dfab124358b8669bb4f01a6dd5
I20251024 19:48:19.930179 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet f6df45dfab124358b8669bb4f01a6dd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:19.950594 1774 ts_tablet_manager.cc:1929] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:19.950817 1774 log.cc:1199] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/d38154d3ffd34f4b8d7305b1c23a4fae
I20251024 19:48:19.951205 1774 ts_tablet_manager.cc:1950] T d38154d3ffd34f4b8d7305b1c23a4fae P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:19.951552 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 849) took 2524 ms. Trace:
I20251024 19:48:19.951702 1774 rpcz_store.cc:276] 1024 19:48:17.427195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.431972 (+ 4777us) service_pool.cc:225] Handling call
1024 19:48:19.848770 (+2416798us) ts_tablet_manager.cc:1151] Deleting tablet d38154d3ffd34f4b8d7305b1c23a4fae
1024 19:48:19.848780 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.922605 (+ 73825us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.950486 (+ 27881us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.951532 (+ 1046us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2416519}
I20251024 19:48:19.952515 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 850) took 2198 ms. Trace:
I20251024 19:48:19.968796 1774 rpcz_store.cc:276] 1024 19:48:17.754567 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.754672 (+ 105us) service_pool.cc:225] Handling call
1024 19:48:19.952358 (+2197686us) ts_tablet_manager.cc:1151] Deleting tablet 94fc8c5716af47faa4e831f05fce8610
1024 19:48:19.952371 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.952497 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2197013}
I20251024 19:48:19.971550 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet e03c488d0f264684b0897a81a04d7466
I20251024 19:48:19.972558 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 854) took 2216 ms. Trace:
I20251024 19:48:19.973601 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet e03c488d0f264684b0897a81a04d7466 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:19.975188 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet f6df45dfab124358b8669bb4f01a6dd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:19.972641 1774 rpcz_store.cc:276] 1024 19:48:17.756101 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.756797 (+ 696us) service_pool.cc:225] Handling call
1024 19:48:19.972407 (+2215610us) ts_tablet_manager.cc:1151] Deleting tablet b6803d23a4714934b85190886611d8d1
1024 19:48:19.972430 (+ 23us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.972538 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2215463}
I20251024 19:48:19.975998 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 855) took 2219 ms. Trace:
I20251024 19:48:19.976325 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet d38154d3ffd34f4b8d7305b1c23a4fae (table default.Table_a395ff27a51a4b5e878e524825bdd2cf [id=ed879934f5754b92b8cff234f3f66333]) successfully deleted
I20251024 19:48:19.976300 1774 rpcz_store.cc:276] 1024 19:48:17.756211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.756969 (+ 758us) service_pool.cc:225] Handling call
1024 19:48:19.975868 (+2218899us) ts_tablet_manager.cc:1151] Deleting tablet 2c96c11697514af6bf4e80a67aadf748
1024 19:48:19.975881 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.975978 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2218703}
W20251024 19:48:19.976593 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 94fc8c5716af47faa4e831f05fce8610 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94fc8c5716af47faa4e831f05fce8610 already in progress: creating tablet
I20251024 19:48:19.976660 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 860) took 2219 ms. Trace:
I20251024 19:48:19.976807 1774 rpcz_store.cc:276] 1024 19:48:17.756745 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.757119 (+ 374us) service_pool.cc:225] Handling call
1024 19:48:19.976542 (+2219423us) ts_tablet_manager.cc:1151] Deleting tablet b724eadbb86e4eeab4ba8dfd565290ff
1024 19:48:19.976553 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.976642 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2219174}
I20251024 19:48:19.977085 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 861) took 2151 ms. Trace:
I20251024 19:48:19.977085 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7f7db12b2ef64686bb2a1b4d4cd7ed67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:19.977916 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet e03c488d0f264684b0897a81a04d7466 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:19.979173 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f7db12b2ef64686bb2a1b4d4cd7ed67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:19.981674 1767 consensus_queue.cc:260] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.981881 3141 raft_consensus.cc:493] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:19.981997 3141 raft_consensus.cc:515] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.982654 3141 leader_election.cc:290] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.983433 1766 tablet_bootstrap.cc:700] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e: Time spent opening tablet: real 0.108s user 0.000s sys 0.000s
I20251024 19:48:19.983759 1766 tablet_bootstrap.cc:654] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:19.984371 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d01572e51694d9bb58c3a59d4205102" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:19.984537 1504 raft_consensus.cc:2468] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:19.985345 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet f6df45dfab124358b8669bb4f01a6dd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:19.977175 1774 rpcz_store.cc:276] 1024 19:48:17.825304 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.825401 (+ 97us) service_pool.cc:225] Handling call
1024 19:48:19.976989 (+2151588us) ts_tablet_manager.cc:1151] Deleting tablet f4e812c43ade4831af50825474976c8e
1024 19:48:19.976997 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.977060 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2151306}
I20251024 19:48:19.985762 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d01572e51694d9bb58c3a59d4205102" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:19.985991 1774 tablet_replica.cc:333] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:19.986222 1774 raft_consensus.cc:2243] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:19.986428 1774 raft_consensus.cc:2272] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:19.986881 1774 ts_tablet_manager.cc:1916] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:19.987043 1586 leader_election.cc:343] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.987438 1586 leader_election.cc:304] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:19.987684 2923 raft_consensus.cc:2804] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:19.987752 2923 raft_consensus.cc:493] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:19.987805 2923 raft_consensus.cc:3060] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.989099 1767 ts_tablet_manager.cc:1434] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.120s user 0.002s sys 0.000s
I20251024 19:48:19.989509 1767 tablet_bootstrap.cc:492] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:19.990677 2923 raft_consensus.cc:515] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:19.990751 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b50a1dbc517648769e34a3655fcef35f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:19.991768 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b1bf5228ae5f485abeb61ea7d512b30d" } new_table_name: "default.Table_7b5a349ef82a44d0861177dc73afad8d" modify_external_catalogs: true
W20251024 19:48:19.993187 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b50a1dbc517648769e34a3655fcef35f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:19.993114 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d01572e51694d9bb58c3a59d4205102" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:19.993283 2923 leader_election.cc:290] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:19.993315 1505 raft_consensus.cc:3060] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:19.993943 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_da7aeba16fdd4a0fa3d17ada53c2f9fb" } new_table_name: "default.Table_68d0a67550994290ae3ea2a721ea6bc7" modify_external_catalogs: true
I20251024 19:48:19.994782 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d01572e51694d9bb58c3a59d4205102" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:19.995226 1586 leader_election.cc:343] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:19.999606 1776 ts_tablet_manager.cc:1929] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:19.999751 1505 raft_consensus.cc:2468] T 2d01572e51694d9bb58c3a59d4205102 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:20.003410 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet e03c488d0f264684b0897a81a04d7466 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:20.003609 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b6803d23a4714934b85190886611d8d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6803d23a4714934b85190886611d8d1 already in progress: creating tablet
W20251024 19:48:20.003940 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2c96c11697514af6bf4e80a67aadf748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c96c11697514af6bf4e80a67aadf748 already in progress: creating tablet
W20251024 19:48:20.004204 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b724eadbb86e4eeab4ba8dfd565290ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b724eadbb86e4eeab4ba8dfd565290ff already in progress: creating tablet
W20251024 19:48:20.004458 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet f4e812c43ade4831af50825474976c8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4e812c43ade4831af50825474976c8e already in progress: creating tablet
I20251024 19:48:20.005084 1766 tablet_bootstrap.cc:492] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
W20251024 19:48:20.014364 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c9ff67824254c04bd7d024cbcc6c080 consensus state has no leader
W20251024 19:48:20.015180 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 19d83464fbf44f78881c7cae43a72091 consensus state has no leader
I20251024 19:48:20.017897 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4c91e1a2322c48b69d7b31b4dbdbac64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:20.018226 1766 ts_tablet_manager.cc:1403] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.144s user 0.004s sys 0.000s
W20251024 19:48:20.018757 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c91e1a2322c48b69d7b31b4dbdbac64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.019510 1766 raft_consensus.cc:359] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.019781 1766 raft_consensus.cc:385] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.019886 1766 raft_consensus.cc:740] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.019937 1768 ts_tablet_manager.cc:1434] T d6de2c21796f4bc4904f88b654c3baec P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.201s user 0.010s sys 0.018s
I20251024 19:48:20.020277 1768 tablet_bootstrap.cc:492] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:20.020202 1766 consensus_queue.cc:260] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.020378 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fe16d951e74c460f901b1898a2a06551"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:20.020864 1766 ts_tablet_manager.cc:1434] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251024 19:48:20.021011 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe16d951e74c460f901b1898a2a06551 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.023523 1187 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 2d01572e51694d9bb58c3a59d4205102 (table default.Table_bf15d3c700774490b1860d3d8865a95e [id=2ea13e78137d47429d7cbe374d2113ef]). Expected version 1 got 0
I20251024 19:48:20.000211 1586 leader_election.cc:304] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:20.027516 2923 raft_consensus.cc:2804] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:20.027750 2923 raft_consensus.cc:697] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:20.028177 2923 consensus_queue.cc:237] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.029099 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a9322f2fceb94f43b0c6e920cc9da52c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.029784 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9322f2fceb94f43b0c6e920cc9da52c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.030826 1776 log.cc:1199] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/16bc737104ab46b3b071a0760dbc847d
I20251024 19:48:20.033402 1775 tablet_replica.cc:333] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:20.033702 1775 raft_consensus.cc:2243] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.033664 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_686d12c4c37f4292aa2c15c0ee048f1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:20.034071 1775 raft_consensus.cc:2272] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:20.034199 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_686d12c4c37f4292aa2c15c0ee048f1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.034909 1775 ts_tablet_manager.cc:1916] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.036713 1186 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet a1fbbba88a294adfbf97c0343d86698e (table default.Table_7bc8cd4a930b44dfafa473297875594d [id=a60c8222aa0248c091fab8dc34100847]). Expected version 2 got 0
I20251024 19:48:20.037007 1186 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (table default.Table_18e7285b7df44399ab3be1dfc86eb91f [id=14bda7f9324f44aa9cb87a5f129039be]). Expected version 2 got 0
I20251024 19:48:20.037380 1186 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet d6de2c21796f4bc4904f88b654c3baec (table default.Table_1a079b981b694617b3baf9897dfb37ae [id=18ec057e920e4c21ae8cbe298058b6f3]). Expected version 2 got 0
I20251024 19:48:20.040076 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet d2a68bb1dc6c483388784b9bb19ce065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.043399 1776 ts_tablet_manager.cc:1950] T 16bc737104ab46b3b071a0760dbc847d P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:20.044365 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 932) took 2501 ms. Trace:
I20251024 19:48:20.044440 1776 rpcz_store.cc:276] 1024 19:48:17.543149 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.543213 (+ 64us) service_pool.cc:225] Handling call
1024 19:48:19.867306 (+2324093us) ts_tablet_manager.cc:1151] Deleting tablet 16bc737104ab46b3b071a0760dbc847d
1024 19:48:19.867317 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:19.916511 (+ 49194us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:19.999489 (+ 82978us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.044347 (+ 44858us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2323892}
I20251024 19:48:20.044847 1776 tablet_replica.cc:333] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:20.045023 1776 raft_consensus.cc:2243] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.045163 1776 raft_consensus.cc:2272] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.045708 1776 ts_tablet_manager.cc:1916] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.046649 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 16bc737104ab46b3b071a0760dbc847d (table default.Table_775e17f58fef4f2b850035a03e411a08 [id=6239ceae506442b79bbe79c48813b49e]) successfully deleted
I20251024 19:48:20.047102 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b71e2a66a1cf4e47bf6cffbbef51f618" } modify_external_catalogs: true
I20251024 19:48:20.047544 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b71e2a66a1cf4e47bf6cffbbef51f618" } modify_external_catalogs: true
I20251024 19:48:20.052795 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet f3d80885fcc642b8a75da30cd4abecbd with delete_type TABLET_DATA_DELETED (replaced by tablet 780442b310e6409498e6bbe4256280a4) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.053036 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 28005301f9de48e5826f46c79aeb5fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.053411 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 1edc55caab904722bf19a1c71df601cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.063628 2923 raft_consensus.cc:493] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.063902 2923 raft_consensus.cc:515] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.065080 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81a3de69a13c459b9b7c02761ab7261e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:20.065270 1505 raft_consensus.cc:2468] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:20.065291 2923 leader_election.cc:290] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.065696 1586 leader_election.cc:304] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:20.007615 1767 tablet_bootstrap.cc:654] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.066058 2923 raft_consensus.cc:2804] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:20.067401 2923 raft_consensus.cc:493] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:20.067550 2923 raft_consensus.cc:3060] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.070611 2923 raft_consensus.cc:515] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.071542 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81a3de69a13c459b9b7c02761ab7261e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:20.071733 1505 raft_consensus.cc:3060] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.071800 2923 leader_election.cc:290] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.073501 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81a3de69a13c459b9b7c02761ab7261e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:20.074081 1586 leader_election.cc:343] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.074488 1505 raft_consensus.cc:2468] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:20.074522 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81a3de69a13c459b9b7c02761ab7261e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:20.075104 1586 leader_election.cc:343] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.075302 1586 leader_election.cc:304] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:20.075767 3141 raft_consensus.cc:2804] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:20.075881 3141 raft_consensus.cc:697] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:20.076108 3141 consensus_queue.cc:237] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.077265 1766 tablet_bootstrap.cc:492] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:20.080232 1186 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet c9cff66282794cc2b6171744a07ea2a6
I20251024 19:48:20.081099 1766 tablet_bootstrap.cc:654] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.082316 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 6b1a3523954448de93357510e6c1ec4e (DEFAULT_TABLE table=default.Table_6bb9ce85a6e341e7b2cc6d9ffc2c0127 [id=50a4f64cc7504ea190832dd265c81caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:20.082227 1186 catalog_manager.cc:5649] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:20.083278 1186 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 2d01572e51694d9bb58c3a59d4205102 (table default.Table_bf15d3c700774490b1860d3d8865a95e [id=2ea13e78137d47429d7cbe374d2113ef]). Expected version 1 got 0
I20251024 19:48:20.083714 2923 raft_consensus.cc:493] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.083877 2923 raft_consensus.cc:515] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.084621 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ae6a9de86a02421a921a5552af7a1e9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:20.084867 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 96a6e0a4882c42399fd07f06ac21aa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.085132 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 65424b00ab9c454f9584048caba83e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:20.085270 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae6a9de86a02421a921a5552af7a1e9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.086182 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d604f7e8552840359eae4a05ca6b65c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:20.087383 2923 leader_election.cc:290] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:20.088701 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d604f7e8552840359eae4a05ca6b65c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.086970 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6424b2347ef642feb38a9596f2d2be1d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:20.089800 1367 raft_consensus.cc:3060] T d6de2c21796f4bc4904f88b654c3baec P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.090041 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 8d1b6eaa353c4284a2b114b42437272c
W20251024 19:48:20.090111 1586 leader_election.cc:343] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.090912 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3ef6d244c498452ca79cd6c6d5c003df
I20251024 19:48:20.090986 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d1b6eaa353c4284a2b114b42437272c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.091387 1192 catalog_manager.cc:6719] ReplaceTablet: tablet 3ef6d244c498452ca79cd6c6d5c003df deleted and replaced by tablet d72f4761820e430cb1a97d83cd30c75d
I20251024 19:48:20.092965 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef6d244c498452ca79cd6c6d5c003df with delete_type TABLET_DATA_DELETED (replaced by tablet d72f4761820e430cb1a97d83cd30c75d) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.092974 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9287f340fb554904986f6c18b7356701
I20251024 19:48:20.093492 1774 ts_tablet_manager.cc:1929] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:20.093538 1191 catalog_manager.cc:6719] ReplaceTablet: tablet 9287f340fb554904986f6c18b7356701 deleted and replaced by tablet 93d1bf52c34c49318de2ca4c27b35cf4
I20251024 19:48:20.093626 1774 log.cc:1199] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/187bc968c2b049028dd25dfb0264c0aa
I20251024 19:48:20.093874 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3c708051e9754fe8afc45154b0a4f30c
I20251024 19:48:20.093936 1774 ts_tablet_manager.cc:1950] T 187bc968c2b049028dd25dfb0264c0aa P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:20.094256 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 864) took 2024 ms. Trace:
I20251024 19:48:20.094336 1774 rpcz_store.cc:276] 1024 19:48:18.070151 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.070206 (+ 55us) service_pool.cc:225] Handling call
1024 19:48:19.985873 (+1915667us) ts_tablet_manager.cc:1151] Deleting tablet 187bc968c2b049028dd25dfb0264c0aa
1024 19:48:19.985947 (+ 74us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.061332 (+ 75385us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.093361 (+ 32029us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.094239 (+ 878us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1915490}
I20251024 19:48:20.094731 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 863) took 2024 ms. Trace:
I20251024 19:48:20.094799 1774 rpcz_store.cc:276] 1024 19:48:18.070040 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.070360 (+ 320us) service_pool.cc:225] Handling call
1024 19:48:20.094595 (+2024235us) ts_tablet_manager.cc:1151] Deleting tablet aeeaf256b1404653b690c712d9ef7a6d
1024 19:48:20.094604 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.094714 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2024072}
I20251024 19:48:20.094906 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3a712f52cd114e9bb07c2717e5d728fd
I20251024 19:48:20.095037 1774 tablet_replica.cc:333] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:20.095212 1774 raft_consensus.cc:2243] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.095418 1185 catalog_manager.cc:6719] ReplaceTablet: tablet 3a712f52cd114e9bb07c2717e5d728fd deleted and replaced by tablet 186bac10f00d4396a3674a9037919e7c
I20251024 19:48:20.095791 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 9287f340fb554904986f6c18b7356701 with delete_type TABLET_DATA_DELETED (replaced by tablet 93d1bf52c34c49318de2ca4c27b35cf4) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.096272 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c708051e9754fe8afc45154b0a4f30c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.096272 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a712f52cd114e9bb07c2717e5d728fd with delete_type TABLET_DATA_DELETED (replaced by tablet 186bac10f00d4396a3674a9037919e7c) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.095374 1774 raft_consensus.cc:2272] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.097146 1774 ts_tablet_manager.cc:1916] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.098408 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d1b6eaa353c4284a2b114b42437272c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.099319 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef6d244c498452ca79cd6c6d5c003df with delete_type TABLET_DATA_DELETED (replaced by tablet d72f4761820e430cb1a97d83cd30c75d) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.100231 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 9287f340fb554904986f6c18b7356701 with delete_type TABLET_DATA_DELETED (replaced by tablet 93d1bf52c34c49318de2ca4c27b35cf4) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.101197 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c708051e9754fe8afc45154b0a4f30c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.102010 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a712f52cd114e9bb07c2717e5d728fd with delete_type TABLET_DATA_DELETED (replaced by tablet 186bac10f00d4396a3674a9037919e7c) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.102686 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b8596dd14d1a4384a3aa199e688330af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.103353 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8596dd14d1a4384a3aa199e688330af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.104946 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3864c444fa71452eb70094031c884901" } new_table_name: "default.Table_8d361d45b27e43d0873aee1e2ebac47a" modify_external_catalogs: true
I20251024 19:48:20.106900 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 6b1a3523954448de93357510e6c1ec4e (DEFAULT_TABLE table=default.Table_6bb9ce85a6e341e7b2cc6d9ffc2c0127 [id=50a4f64cc7504ea190832dd265c81caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:20.107201 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ef6d244c498452ca79cd6c6d5c003df with delete_type TABLET_DATA_DELETED (replaced by tablet d72f4761820e430cb1a97d83cd30c75d) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.107276 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1a3523954448de93357510e6c1ec4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:20.107563 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 187bc968c2b049028dd25dfb0264c0aa (table default.Table_31fb0d4536b94892aeceb91307c74837 [id=2bd12943c3e24d70a4f02626d7d58f6c]) successfully deleted
W20251024 19:48:20.107851 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet aeeaf256b1404653b690c712d9ef7a6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aeeaf256b1404653b690c712d9ef7a6d already in progress: creating tablet
I20251024 19:48:20.107983 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d1b6eaa353c4284a2b114b42437272c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.108155 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a712f52cd114e9bb07c2717e5d728fd with delete_type TABLET_DATA_DELETED (replaced by tablet 186bac10f00d4396a3674a9037919e7c) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.108343 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c708051e9754fe8afc45154b0a4f30c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.108546 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 9287f340fb554904986f6c18b7356701 with delete_type TABLET_DATA_DELETED (replaced by tablet 93d1bf52c34c49318de2ca4c27b35cf4) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.109755 1775 ts_tablet_manager.cc:1929] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.109967 1775 log.cc:1199] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/f83b9290f00145a780318a43249eec12
I20251024 19:48:20.111163 1767 tablet_bootstrap.cc:492] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:20.111377 1767 ts_tablet_manager.cc:1403] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.122s user 0.005s sys 0.000s
I20251024 19:48:20.112993 1767 raft_consensus.cc:359] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.113336 1767 raft_consensus.cc:385] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.113492 1767 raft_consensus.cc:740] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.113926 1767 consensus_queue.cc:260] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.115195 1767 ts_tablet_manager.cc:1434] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20251024 19:48:20.109351 1366 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.005s sys 0.033s
W20251024 19:48:20.115516 1366 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s user 0.005s sys 0.033s
I20251024 19:48:20.115868 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f04fa9e7d054461894b1ded30ab195a1" } new_table_name: "default.Table_68151dcee0bb49ed8a780b240265fc91" modify_external_catalogs: true
I20251024 19:48:20.117740 3210 consensus_queue.cc:1048] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:20.122887 1767 tablet_bootstrap.cc:492] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:20.125356 1767 tablet_bootstrap.cc:654] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.125764 2936 consensus_queue.cc:1048] T 067631b128fd4a1e9ab72c9507dab6c4 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:20.126999 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_93d0f441b9cb47e9a13b6cb7f04c4ac9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.127625 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93d0f441b9cb47e9a13b6cb7f04c4ac9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.129789 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_98740f996f06427dbd2383bf1c19bc1b" } new_table_name: "default.Table_9f07b2d2fd0e4c88a680f1e517424bc6" modify_external_catalogs: true
I20251024 19:48:20.132027 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 6b1a3523954448de93357510e6c1ec4e (DEFAULT_TABLE table=default.Table_6bb9ce85a6e341e7b2cc6d9ffc2c0127 [id=50a4f64cc7504ea190832dd265c81caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:20.132416 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1a3523954448de93357510e6c1ec4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:20.133945 1190 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 067631b128fd4a1e9ab72c9507dab6c4 (table default.Table_814a6dad46d54945b70e652b7e0f0032 [id=3f394c59d82b44928beb22f42312a4a3]). Expected version 1 got 0
I20251024 19:48:20.116246 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6424b2347ef642feb38a9596f2d2be1d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:20.135293 1505 raft_consensus.cc:2468] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:20.122570 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1a3523954448de93357510e6c1ec4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:20.142714 1586 leader_election.cc:304] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:20.143415 2923 raft_consensus.cc:2804] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:20.143515 2923 raft_consensus.cc:493] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:20.143581 2923 raft_consensus.cc:3060] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.147361 3137 consensus_queue.cc:1048] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:20.151810 2923 raft_consensus.cc:515] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.152900 2923 leader_election.cc:290] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.153337 1774 ts_tablet_manager.cc:1929] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.153539 1774 log.cc:1199] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/3887fc21726846279fecb02672b9bc07
I20251024 19:48:20.154270 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6424b2347ef642feb38a9596f2d2be1d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:20.154465 1504 raft_consensus.cc:3060] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.157222 1504 raft_consensus.cc:2468] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:20.157795 2858 raft_consensus.cc:493] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.157925 2858 raft_consensus.cc:515] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.158526 2858 leader_election.cc:290] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.159013 1586 leader_election.cc:304] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:20.159065 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96a6e0a4882c42399fd07f06ac21aa3f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:20.159143 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96a6e0a4882c42399fd07f06ac21aa3f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:20.159516 1444 leader_election.cc:343] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.159672 3141 raft_consensus.cc:2804] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:20.159806 3141 raft_consensus.cc:697] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
W20251024 19:48:20.159824 1447 leader_election.cc:343] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.159924 1447 leader_election.cc:304] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:20.160037 3141 consensus_queue.cc:237] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.160394 2858 raft_consensus.cc:2749] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:20.153801 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6424b2347ef642feb38a9596f2d2be1d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:20.175451 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 332d8bd38f0b478d9592c781dfd5ca72 consensus state has no leader
W20251024 19:48:20.175854 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d123756f164f44168200b60bf7c615ca consensus state has no leader
W20251024 19:48:20.176452 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eee85800b6424ad489cdf7325594b71c consensus state has no leader
I20251024 19:48:20.177837 1775 ts_tablet_manager.cc:1950] T f83b9290f00145a780318a43249eec12 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:20.178224 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 882) took 3203 ms. Trace:
I20251024 19:48:20.178314 1775 rpcz_store.cc:276] 1024 19:48:16.975051 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:16.977630 (+ 2579us) service_pool.cc:225] Handling call
1024 19:48:19.909595 (+2931965us) ts_tablet_manager.cc:1151] Deleting tablet f83b9290f00145a780318a43249eec12
1024 19:48:20.033349 (+123754us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.051294 (+ 17945us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.109637 (+ 58343us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.178205 (+ 68568us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":838,"tablet-delete.queue_time_us":2931713}
I20251024 19:48:20.178627 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6bb9ce85a6e341e7b2cc6d9ffc2c0127" } modify_external_catalogs: true
I20251024 19:48:20.178784 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 891) took 3163 ms. Trace:
I20251024 19:48:20.178831 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6bb9ce85a6e341e7b2cc6d9ffc2c0127" } modify_external_catalogs: true
I20251024 19:48:20.178855 1775 rpcz_store.cc:276] 1024 19:48:17.014929 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.014988 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:20.178637 (+3163649us) ts_tablet_manager.cc:1151] Deleting tablet 3b8e8dbca79f4d72a2de51f105b209e4
1024 19:48:20.178646 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.178766 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3096973}
I20251024 19:48:20.179174 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 892) took 3164 ms. Trace:
I20251024 19:48:20.179241 1775 rpcz_store.cc:276] 1024 19:48:17.015197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.015244 (+ 47us) service_pool.cc:225] Handling call
1024 19:48:20.179072 (+3163828us) ts_tablet_manager.cc:1151] Deleting tablet 0143a393711446fdbe6c798efa0c8936
1024 19:48:20.179082 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.179159 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3097343}
I20251024 19:48:20.179544 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 893) took 3164 ms. Trace:
I20251024 19:48:20.179610 1775 rpcz_store.cc:276] 1024 19:48:17.015451 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.015497 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:20.179421 (+3163924us) ts_tablet_manager.cc:1151] Deleting tablet 4f00719c20934655bc744446194f7401
1024 19:48:20.179430 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.179529 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3097643}
I20251024 19:48:20.179889 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 895) took 3163 ms. Trace:
I20251024 19:48:20.179952 1775 rpcz_store.cc:276] 1024 19:48:17.016371 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.023411 (+ 7040us) service_pool.cc:225] Handling call
1024 19:48:20.179789 (+3156378us) ts_tablet_manager.cc:1151] Deleting tablet 4d0e4ffcdf184c5bafd99bf721b35951
1024 19:48:20.179797 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.179874 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3097968}
I20251024 19:48:20.180238 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 898) took 3146 ms. Trace:
I20251024 19:48:20.180305 1775 rpcz_store.cc:276] 1024 19:48:17.033623 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.033692 (+ 69us) service_pool.cc:225] Handling call
1024 19:48:20.180130 (+3146438us) ts_tablet_manager.cc:1151] Deleting tablet ca6b8783039a4f5fbbda65c129805642
1024 19:48:20.180138 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.180223 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3094453}
I20251024 19:48:20.180579 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 908) took 3031 ms. Trace:
I20251024 19:48:20.180644 1775 rpcz_store.cc:276] 1024 19:48:17.148621 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.148671 (+ 50us) service_pool.cc:225] Handling call
1024 19:48:20.180495 (+3031824us) ts_tablet_manager.cc:1151] Deleting tablet 6a7e76a3cac443ec8d2ab466075aae6b
1024 19:48:20.180504 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.180565 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3031649}
I20251024 19:48:20.180869 1775 tablet_replica.cc:333] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:20.180950 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b5a66c4e9cd8408782690b66cdb87473" } modify_external_catalogs: true
I20251024 19:48:20.181066 1775 raft_consensus.cc:2243] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.181107 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b5a66c4e9cd8408782690b66cdb87473" } modify_external_catalogs: true
I20251024 19:48:20.181236 1775 raft_consensus.cc:2272] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.183852 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet f83b9290f00145a780318a43249eec12 (table default.Table_420ebc050cf440e9a59a18e26f7b83af [id=be1cc9d047554ea6a9afa531fb08a81d]) successfully deleted
W20251024 19:48:20.184108 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3b8e8dbca79f4d72a2de51f105b209e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b8e8dbca79f4d72a2de51f105b209e4 already in progress: creating tablet
W20251024 19:48:20.184341 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 0143a393711446fdbe6c798efa0c8936 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0143a393711446fdbe6c798efa0c8936 already in progress: creating tablet
W20251024 19:48:20.184556 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4f00719c20934655bc744446194f7401 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f00719c20934655bc744446194f7401 already in progress: creating tablet
W20251024 19:48:20.184751 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4d0e4ffcdf184c5bafd99bf721b35951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d0e4ffcdf184c5bafd99bf721b35951 already in progress: creating tablet
W20251024 19:48:20.184973 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet ca6b8783039a4f5fbbda65c129805642 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca6b8783039a4f5fbbda65c129805642 already in progress: creating tablet
W20251024 19:48:20.185177 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 6a7e76a3cac443ec8d2ab466075aae6b because the tablet was not found. No further retry: Not found: Tablet not found: 6a7e76a3cac443ec8d2ab466075aae6b
W20251024 19:48:20.191742 1586 leader_election.cc:343] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.196945 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ee30e626b58d40d294af6f355ea32931"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:20.198786 2923 raft_consensus.cc:493] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.198930 2923 raft_consensus.cc:515] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.199018 2858 raft_consensus.cc:493] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.199134 2858 raft_consensus.cc:515] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.199458 1768 tablet_bootstrap.cc:654] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.199550 2923 leader_election.cc:290] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.199666 2858 leader_election.cc:290] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:20.200131 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee30e626b58d40d294af6f355ea32931 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.200201 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1fbbba88a294adfbf97c0343d86698e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:20.200516 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6db58fac5e4af9b27bde6c223aa845" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:20.200718 1444 leader_election.cc:343] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.201030 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6db58fac5e4af9b27bde6c223aa845" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:20.201200 1504 raft_consensus.cc:2468] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:20.201352 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1fbbba88a294adfbf97c0343d86698e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:20.201805 1447 leader_election.cc:343] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.201915 1447 leader_election.cc:304] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:20.201985 1586 leader_election.cc:304] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:20.202263 2858 raft_consensus.cc:2749] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:20.202440 1586 leader_election.cc:343] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.202808 2923 raft_consensus.cc:2804] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:20.202940 2923 raft_consensus.cc:493] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:20.203064 2923 raft_consensus.cc:3060] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.204731 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 81a3de69a13c459b9b7c02761ab7261e
I20251024 19:48:20.205673 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 6b1a3523954448de93357510e6c1ec4e
I20251024 19:48:20.205760 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 81a3de69a13c459b9b7c02761ab7261e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.206111 2923 raft_consensus.cc:515] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.206758 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b1a3523954448de93357510e6c1ec4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.207034 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6db58fac5e4af9b27bde6c223aa845" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:20.207211 1504 raft_consensus.cc:3060] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.207340 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b1a3523954448de93357510e6c1ec4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.207656 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b1a3523954448de93357510e6c1ec4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.209771 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 81a3de69a13c459b9b7c02761ab7261e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.210348 1504 raft_consensus.cc:2468] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:20.210790 1586 leader_election.cc:304] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:20.211066 3137 raft_consensus.cc:2804] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:20.211170 3137 raft_consensus.cc:697] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:20.211408 3137 consensus_queue.cc:237] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.213177 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 81a3de69a13c459b9b7c02761ab7261e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.215677 1768 tablet_bootstrap.cc:492] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:20.215823 1768 ts_tablet_manager.cc:1403] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.196s user 0.000s sys 0.004s
I20251024 19:48:20.216418 2923 leader_election.cc:290] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.216624 1768 raft_consensus.cc:359] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:20.216784 1768 raft_consensus.cc:385] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.216825 1768 raft_consensus.cc:740] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.217144 1768 consensus_queue.cc:260] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:20.217901 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_accb35a3be634901b43dbbdf579098cd" } new_table_name: "default.Table_d6c796b5abfe4ba0b02c8b2e41101f45" modify_external_catalogs: true
I20251024 19:48:20.218075 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6db58fac5e4af9b27bde6c223aa845" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:20.218813 1768 ts_tablet_manager.cc:1434] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20251024 19:48:20.218556 1586 leader_election.cc:343] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.219340 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_358a359f16ae4eeeb2972452b92e532d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.222079 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_358a359f16ae4eeeb2972452b92e532d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.226572 3160 raft_consensus.cc:493] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af3952de050140d383f6b0de7666fa97)
I20251024 19:48:20.226673 1191 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet d6de2c21796f4bc4904f88b654c3baec (table default.Table_1a079b981b694617b3baf9897dfb37ae [id=18ec057e920e4c21ae8cbe298058b6f3]). Expected version 2 got 0
I20251024 19:48:20.226711 3160 raft_consensus.cc:515] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.227607 3160 leader_election.cc:290] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:20.228904 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59eca6bb1dd84c67a7e53131abaa937a consensus state has no leader
I20251024 19:48:20.229751 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f89a3bbeb9b4d57a49078729110d951" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:20.232318 1766 tablet_bootstrap.cc:492] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:20.232483 1766 ts_tablet_manager.cc:1403] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.156s user 0.007s sys 0.000s
I20251024 19:48:20.235801 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet d6de2c21796f4bc4904f88b654c3baec (table default.Table_1a079b981b694617b3baf9897dfb37ae [id=18ec057e920e4c21ae8cbe298058b6f3]). Expected version 2 got 0
I20251024 19:48:20.238345 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 067631b128fd4a1e9ab72c9507dab6c4 (table default.Table_814a6dad46d54945b70e652b7e0f0032 [id=3f394c59d82b44928beb22f42312a4a3]). Expected version 1 got 0
I20251024 19:48:20.240236 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d129bdc876e425aabcf1c99997e319a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.240737 1768 tablet_bootstrap.cc:492] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:20.219103 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b733659f0fe548d78e86ae0925c680a4" } new_table_name: "default.Table_e506c24615f3483583b10487ce001fd1" modify_external_catalogs: true
I20251024 19:48:20.242219 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f89a3bbeb9b4d57a49078729110d951" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:20.247217 2660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45498 (request call id 0) took 7587 ms. Trace:
I20251024 19:48:20.253671 1776 ts_tablet_manager.cc:1929] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.253844 1776 log.cc:1199] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/b240965605904533bd2af99f61aaef26
I20251024 19:48:20.247978 3061 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc8f71a5b70749a4a3e3a599bcb3d22c. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:48:20.254081 3061 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 19:48:20.248227 3115 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc8f71a5b70749a4a3e3a599bcb3d22c)
I20251024 19:48:20.254236 1776 ts_tablet_manager.cc:1950] T b240965605904533bd2af99f61aaef26 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:20.254421 3115 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } }
I20251024 19:48:20.258459 3115 leader_election.cc:290] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82e82847f8904b59b03f446b7fce6233 (127.0.252.253:41375), dc8f71a5b70749a4a3e3a599bcb3d22c (127.0.252.252:36627)
I20251024 19:48:20.259390 1766 raft_consensus.cc:359] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.259630 1766 raft_consensus.cc:385] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.259694 1766 raft_consensus.cc:740] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.260318 1775 ts_tablet_manager.cc:1916] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.266263 1767 tablet_bootstrap.cc:492] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:20.266518 1767 ts_tablet_manager.cc:1403] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.144s user 0.005s sys 0.001s
I20251024 19:48:20.267410 1767 raft_consensus.cc:359] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.260649 1766 consensus_queue.cc:260] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.270282 1766 ts_tablet_manager.cc:1434] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.038s user 0.002s sys 0.001s
I20251024 19:48:20.271266 1766 tablet_bootstrap.cc:492] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:20.284171 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6237c63465cc4c8daa9136ba2d3a39fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.284888 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6237c63465cc4c8daa9136ba2d3a39fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.287559 1767 raft_consensus.cc:385] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.288686 1767 raft_consensus.cc:740] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.282115 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 933) took 2738 ms. Trace:
I20251024 19:48:20.289103 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1db3c2ecc936440985b132c9db2c56e5" } modify_external_catalogs: true
I20251024 19:48:20.289186 1776 rpcz_store.cc:276] 1024 19:48:17.543447 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.543577 (+ 130us) service_pool.cc:225] Handling call
1024 19:48:20.044805 (+2501228us) ts_tablet_manager.cc:1151] Deleting tablet b240965605904533bd2af99f61aaef26
1024 19:48:20.044814 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.049197 (+ 4383us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.195023 (+145826us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.282081 (+ 87058us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2501077}
I20251024 19:48:20.289414 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1db3c2ecc936440985b132c9db2c56e5" } modify_external_catalogs: true
I20251024 19:48:20.289738 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet b240965605904533bd2af99f61aaef26 (table default.Table_a1366192b3134e3abfc231729cb42344 [id=ae6b320a09934ac5873d3b35c33f2c65]) successfully deleted
I20251024 19:48:20.289844 1776 tablet_replica.cc:333] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:20.290100 1776 raft_consensus.cc:2243] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:20.290174 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_67e6d3d8a4334247a1bf237d5936799b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:20.290531 1776 raft_consensus.cc:2272] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:20.290668 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67e6d3d8a4334247a1bf237d5936799b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.290971 1767 consensus_queue.cc:260] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.291985 1767 ts_tablet_manager.cc:1434] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20251024 19:48:20.242820 1308 leader_election.cc:304] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
W20251024 19:48:20.293058 1308 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 50269.7us
I20251024 19:48:20.294081 1767 tablet_bootstrap.cc:492] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:20.252182 1418 heartbeater.cc:344] Connected to a master server at 127.0.252.254:36505
W20251024 19:48:20.294409 1586 consensus_peers.cc:597] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:20.296056 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 1999b751e9344dcf81d26226c0589c8f (table default.Table_9147a98b3bf9483ea7b7a6f17130d4fc [id=d743705c21da4b89abedf2eb34ddbe09]). Expected version 1 got 0
I20251024 19:48:20.298032 3208 raft_consensus.cc:2749] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:48:20.273085 1364 raft_consensus.cc:3060] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.301458 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a1a3ffe843e4470aa847383e6f72077b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.302201 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1a3ffe843e4470aa847383e6f72077b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.293310 1776 ts_tablet_manager.cc:1916] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.308786 1766 tablet_bootstrap.cc:654] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.309945 1774 ts_tablet_manager.cc:1950] T 3887fc21726846279fecb02672b9bc07 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:20.310391 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 865) took 2239 ms. Trace:
I20251024 19:48:20.310582 1774 rpcz_store.cc:276] 1024 19:48:18.071203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.072017 (+ 814us) service_pool.cc:225] Handling call
1024 19:48:20.094996 (+2022979us) ts_tablet_manager.cc:1151] Deleting tablet 3887fc21726846279fecb02672b9bc07
1024 19:48:20.095005 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.104521 (+ 9516us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.153200 (+ 48679us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.310373 (+157173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2017203}
I20251024 19:48:20.311674 1774 tablet_replica.cc:333] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:20.311910 1774 raft_consensus.cc:2243] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.312132 1774 raft_consensus.cc:2272] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.314935 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 3887fc21726846279fecb02672b9bc07 (table default.Table_e378095ddecc4dcf801a1568160c34b4 [id=450b69bd10cf4c629cc3f8dadc790312]) successfully deleted
I20251024 19:48:20.315964 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_50e12cf5f39d4fa2983835ef0dbbc3d6" } modify_external_catalogs: true
I20251024 19:48:20.316221 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_50e12cf5f39d4fa2983835ef0dbbc3d6" } modify_external_catalogs: true
I20251024 19:48:20.317775 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2b311ba111854dabaeaaff96e18ee699" } modify_external_catalogs: true
I20251024 19:48:20.317958 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2b311ba111854dabaeaaff96e18ee699" } modify_external_catalogs: true
I20251024 19:48:20.318562 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_038012ef539b428f9984af28f5dec4c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.319275 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_038012ef539b428f9984af28f5dec4c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.321130 2858 raft_consensus.cc:493] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.321307 2858 raft_consensus.cc:515] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.321992 2858 leader_election.cc:290] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.325606 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eafde5a40df4fbfb03340b7bf791a2a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:20.326169 1447 leader_election.cc:343] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.328531 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eafde5a40df4fbfb03340b7bf791a2a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:20.328980 1774 ts_tablet_manager.cc:1916] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:20.329078 1444 leader_election.cc:343] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.329187 1444 leader_election.cc:304] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:20.329689 2858 raft_consensus.cc:2749] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:20.335230 1767 tablet_bootstrap.cc:654] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.336795 1768 tablet_bootstrap.cc:654] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.342429 1766 tablet_bootstrap.cc:492] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:20.342602 1766 ts_tablet_manager.cc:1403] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.002s
I20251024 19:48:20.343379 1766 raft_consensus.cc:359] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.343572 1766 raft_consensus.cc:385] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.343631 1766 raft_consensus.cc:740] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.343923 1766 consensus_queue.cc:260] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.344568 1766 ts_tablet_manager.cc:1434] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:20.344866 1766 tablet_bootstrap.cc:492] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:20.354857 1766 tablet_bootstrap.cc:654] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.362017 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 2d01572e51694d9bb58c3a59d4205102 (table default.Table_bf15d3c700774490b1860d3d8865a95e [id=2ea13e78137d47429d7cbe374d2113ef]). Expected version 1 got 0
I20251024 19:48:20.362603 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 067631b128fd4a1e9ab72c9507dab6c4 (table default.Table_814a6dad46d54945b70e652b7e0f0032 [id=3f394c59d82b44928beb22f42312a4a3]). Expected version 1 got 0
W20251024 19:48:20.358294 1586 consensus_peers.cc:597] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:20.364255 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e6db58fac5e4af9b27bde6c223aa845 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.381354 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_00523012c9d142bc81e566a6e515e07f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:20.381704 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_dab0576c96374dd0b1cd1409c1594039"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.382050 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00523012c9d142bc81e566a6e515e07f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:20.382139 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dab0576c96374dd0b1cd1409c1594039 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.387378 1768 tablet_bootstrap.cc:492] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:20.387534 1768 ts_tablet_manager.cc:1403] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.147s user 0.005s sys 0.001s
I20251024 19:48:20.389884 1364 raft_consensus.cc:3060] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.390029 3137 consensus_queue.cc:1048] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:20.392544 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f65ea29d17fe40599ad9377f12464b7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.393261 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f65ea29d17fe40599ad9377f12464b7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.395004 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d7ded51d900c4f8cabddfdb0f3d069ec" } new_table_name: "default.Table_af1d07fdb0de4e7aaf00ac9a44631f19" modify_external_catalogs: true
I20251024 19:48:20.396567 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b20a6845ae1645c0b51f096d471d4ccc
I20251024 19:48:20.397306 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d494ae379f1a42cabbe4ea77a4668e45" } new_table_name: "default.Table_44ebe3ca730f4ede80fae25d92a70383" modify_external_catalogs: true
I20251024 19:48:20.398803 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3b12fc2aa8564985980b459691c71555"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.399314 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b12fc2aa8564985980b459691c71555 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.401845 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet b20a6845ae1645c0b51f096d471d4ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.404232 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet b20a6845ae1645c0b51f096d471d4ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.404372 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet b20a6845ae1645c0b51f096d471d4ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.405094 1774 ts_tablet_manager.cc:1929] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.405404 1767 tablet_bootstrap.cc:492] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:20.406888 1775 ts_tablet_manager.cc:1929] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.406955 1774 log.cc:1199] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/be8126459ef541f39bc133f27c7d0f59
I20251024 19:48:20.407408 1767 ts_tablet_manager.cc:1403] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.113s user 0.003s sys 0.003s
I20251024 19:48:20.414041 1774 ts_tablet_manager.cc:1950] T be8126459ef541f39bc133f27c7d0f59 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:48:20.411638 1586 consensus_peers.cc:597] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:20.415973 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:20.416114 1767 raft_consensus.cc:359] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:20.416296 1767 raft_consensus.cc:385] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.416361 1767 raft_consensus.cc:740] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
W20251024 19:48:20.416477 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4a66b7c89794807826bea29229a6d3a consensus state has no leader
I20251024 19:48:20.416608 1767 consensus_queue.cc:260] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:20.416965 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_039c8e8c0a8946a59f10e06691e5b946" } new_table_name: "default.Table_3ae09b04a7ef498b9ebe1867f056d9c9" modify_external_catalogs: true
I20251024 19:48:20.417094 1767 ts_tablet_manager.cc:1434] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:20.417352 1767 tablet_bootstrap.cc:492] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:20.420519 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet fc187c8f34594f4fb37d8afe0855263b
I20251024 19:48:20.421563 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ea0d3589c13a4ebeb950ddc737a9948b" } modify_external_catalogs: true
I20251024 19:48:20.421499 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet fc187c8f34594f4fb37d8afe0855263b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.421792 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ea0d3589c13a4ebeb950ddc737a9948b" } modify_external_catalogs: true
I20251024 19:48:20.422827 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet fc187c8f34594f4fb37d8afe0855263b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.425673 1775 log.cc:1199] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/2602335327fc489fa154056c5ef5138f
I20251024 19:48:20.426149 1775 ts_tablet_manager.cc:1950] T 2602335327fc489fa154056c5ef5138f P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:20.426290 2858 raft_consensus.cc:493] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af3952de050140d383f6b0de7666fa97)
I20251024 19:48:20.426522 2858 raft_consensus.cc:515] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.427053 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 907) took 3278 ms. Trace:
I20251024 19:48:20.427170 2858 leader_election.cc:290] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.408205 1766 tablet_bootstrap.cc:492] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:20.414570 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 0266cef4e2034303978d7456c47ea98c with delete_type TABLET_DATA_DELETED (replaced by tablet 64325d89db714554866e74922d103e34) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.423424 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet fc187c8f34594f4fb37d8afe0855263b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.427390 1766 ts_tablet_manager.cc:1403] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.083s user 0.001s sys 0.005s
W20251024 19:48:20.428108 1444 leader_election.cc:343] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: e29abfa3bc8944aea7d1acd03e3eb897
I20251024 19:48:20.428186 1766 raft_consensus.cc:359] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.428339 1766 raft_consensus.cc:385] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.428416 1766 raft_consensus.cc:740] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.428638 1766 consensus_queue.cc:260] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.429352 1766 ts_tablet_manager.cc:1434] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:20.430100 2857 raft_consensus.cc:493] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.430215 2857 raft_consensus.cc:515] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.430768 2857 leader_election.cc:290] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.431358 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1999b751e9344dcf81d26226c0589c8f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:20.431896 1444 leader_election.cc:343] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.432816 3163 raft_consensus.cc:493] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.432940 3163 raft_consensus.cc:515] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.433310 1766 tablet_bootstrap.cc:492] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:20.433506 3163 leader_election.cc:290] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.434280 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1999b751e9344dcf81d26226c0589c8f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:20.434337 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50cea450eda9485daba40bf875a1b90c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:20.434813 1447 leader_election.cc:343] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.434948 1447 leader_election.cc:304] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:20.435002 1586 leader_election.cc:343] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.435287 2857 raft_consensus.cc:2749] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:20.435611 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50cea450eda9485daba40bf875a1b90c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:20.435832 1505 raft_consensus.cc:2468] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:20.436280 1586 leader_election.cc:304] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:20.437341 3137 raft_consensus.cc:2804] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:20.437438 3137 raft_consensus.cc:493] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:20.437503 3137 raft_consensus.cc:3060] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:20.438836 1447 leader_election.cc:343] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: e29abfa3bc8944aea7d1acd03e3eb897
I20251024 19:48:20.438941 1447 leader_election.cc:304] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:20.440352 3137 raft_consensus.cc:515] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.440769 2857 raft_consensus.cc:2749] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:48:20.440913 3137 leader_election.cc:290] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.440892 1768 raft_consensus.cc:359] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.441064 1768 raft_consensus.cc:385] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.441124 1768 raft_consensus.cc:740] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.441341 1768 consensus_queue.cc:260] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.442041 1768 ts_tablet_manager.cc:1434] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.054s user 0.001s sys 0.001s
I20251024 19:48:20.442333 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50cea450eda9485daba40bf875a1b90c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:20.442512 1511 raft_consensus.cc:3060] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.442692 1766 tablet_bootstrap.cc:654] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.445319 1511 raft_consensus.cc:2468] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:20.427136 1775 rpcz_store.cc:276] 1024 19:48:17.148345 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.148533 (+ 188us) service_pool.cc:225] Handling call
1024 19:48:20.180825 (+3032292us) ts_tablet_manager.cc:1151] Deleting tablet 2602335327fc489fa154056c5ef5138f
1024 19:48:20.180835 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.306900 (+126065us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.406745 (+ 99845us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.427034 (+ 20289us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3031536}
I20251024 19:48:20.446415 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 873) took 2337 ms. Trace:
I20251024 19:48:20.446538 1774 rpcz_store.cc:276] 1024 19:48:18.109286 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.109830 (+ 544us) service_pool.cc:225] Handling call
1024 19:48:20.311628 (+2201798us) ts_tablet_manager.cc:1151] Deleting tablet be8126459ef541f39bc133f27c7d0f59
1024 19:48:20.311638 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.332446 (+ 20808us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.404980 (+ 72534us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.446396 (+ 41416us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2196376}
I20251024 19:48:20.447017 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet be8126459ef541f39bc133f27c7d0f59 (table default.Table_0d796a82cf4b4e0f9cba35051c69293d [id=06db25030489409aa797ab8ea05dd6c5]) successfully deleted
I20251024 19:48:20.447157 1586 leader_election.cc:304] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:20.447590 3141 raft_consensus.cc:2804] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:20.447705 3141 raft_consensus.cc:697] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:20.447944 3141 consensus_queue.cc:237] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.447470 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 874) took 2266 ms. Trace:
I20251024 19:48:20.448199 1774 rpcz_store.cc:276] 1024 19:48:18.180948 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.181876 (+ 928us) service_pool.cc:225] Handling call
1024 19:48:20.447326 (+2265450us) ts_tablet_manager.cc:1151] Deleting tablet e7d6472ac7b046d89f56618fa4ed1811
1024 19:48:20.447338 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.447450 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2265253}
W20251024 19:48:20.448621 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet e7d6472ac7b046d89f56618fa4ed1811 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7d6472ac7b046d89f56618fa4ed1811 already in progress: creating tablet
I20251024 19:48:20.449126 1774 tablet_replica.cc:333] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:20.453295 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2461bf28f0bd44ac8e86c5bd843a6718" } modify_external_catalogs: true
I20251024 19:48:20.453477 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2461bf28f0bd44ac8e86c5bd843a6718" } modify_external_catalogs: true
W20251024 19:48:20.453859 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec0a8fc1b67b4314ad8d305a9d13a1b6 consensus state has no leader
I20251024 19:48:20.453910 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet ee68243b4a664575b679950603bb250e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.454233 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_dd0f43494b6b45c1b34861a14440b2eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.454728 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd0f43494b6b45c1b34861a14440b2eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:20.456248 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4229bd266e04d04b970bc9fc760155f consensus state has no leader
I20251024 19:48:20.458673 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e2e3a2eac58e42fa8d80c405e3ec136a" } new_table_name: "default.Table_a1ee0f469ecd4399be90ed862b1b188a" modify_external_catalogs: true
I20251024 19:48:20.460340 1776 ts_tablet_manager.cc:1929] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.460451 1776 log.cc:1199] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/be8126459ef541f39bc133f27c7d0f59
I20251024 19:48:20.460780 1776 ts_tablet_manager.cc:1950] T be8126459ef541f39bc133f27c7d0f59 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
W20251024 19:48:20.466559 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44305f9f044b4b14bbf66bbe612d14a1 consensus state has no leader
I20251024 19:48:20.467231 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet a3259823b622461fa823841925b3d203
I20251024 19:48:20.468513 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet a3259823b622461fa823841925b3d203 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.469115 1767 tablet_bootstrap.cc:654] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.478119 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 934) took 2934 ms. Trace:
I20251024 19:48:20.478734 1776 rpcz_store.cc:276] 1024 19:48:17.543676 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.543743 (+ 67us) service_pool.cc:225] Handling call
1024 19:48:20.289776 (+2746033us) ts_tablet_manager.cc:1151] Deleting tablet be8126459ef541f39bc133f27c7d0f59
1024 19:48:20.289791 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.384888 (+ 95097us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.460223 (+ 75335us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.478091 (+ 17868us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2745827}
I20251024 19:48:20.490563 1776 tablet_replica.cc:333] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:20.490612 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet be8126459ef541f39bc133f27c7d0f59 (table default.Table_0d796a82cf4b4e0f9cba35051c69293d [id=06db25030489409aa797ab8ea05dd6c5]) successfully deleted
I20251024 19:48:20.491405 1776 raft_consensus.cc:2243] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:20.491825 1776 raft_consensus.cc:2272] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.492234 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4d52c098fe604c2f885442d39f4238b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.493014 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d52c098fe604c2f885442d39f4238b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.497138 1776 ts_tablet_manager.cc:1916] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.513084 2604 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "af3952de050140d383f6b0de7666fa97" instance_seqno: 1761335275140404) as {username='slave'} at 127.0.252.195:45731; Asking this server to re-register.
I20251024 19:48:20.514271 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a1031d785b646b095ece9be666d2a77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.519202 1696 heartbeater.cc:461] Registering TS with master...
W20251024 19:48:20.520376 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 563897855a6e4c23803bf4ae34115850 consensus state has no leader
I20251024 19:48:20.449657 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50cea450eda9485daba40bf875a1b90c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:20.470448 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 844638a6b6944d0394df3e78a8ea51d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.490445 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet a3259823b622461fa823841925b3d203 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.449352 1774 raft_consensus.cc:2243] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.522372 1774 raft_consensus.cc:2272] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.523039 1774 ts_tablet_manager.cc:1916] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.523464 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet a2cc3f3d02324c1e867b8fc159ad6d20
W20251024 19:48:20.524446 1586 leader_election.cc:343] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.525516 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet a2cc3f3d02324c1e867b8fc159ad6d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:20.499014 1768 tablet_bootstrap.cc:492] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:20.527293 1767 tablet_bootstrap.cc:492] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:20.527443 1767 ts_tablet_manager.cc:1403] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.110s user 0.002s sys 0.002s
I20251024 19:48:20.528796 1767 raft_consensus.cc:359] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.528985 1767 raft_consensus.cc:385] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.529062 1767 raft_consensus.cc:740] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.529306 1767 consensus_queue.cc:260] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.532298 1767 ts_tablet_manager.cc:1434] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20251024 19:48:20.534091 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet a2cc3f3d02324c1e867b8fc159ad6d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.537091 1776 ts_tablet_manager.cc:1929] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.537273 1776 log.cc:1199] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/2602335327fc489fa154056c5ef5138f
I20251024 19:48:20.537719 1776 ts_tablet_manager.cc:1950] T 2602335327fc489fa154056c5ef5138f P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:20.548967 3141 raft_consensus.cc:493] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.549187 3141 raft_consensus.cc:515] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.549919 3141 leader_election.cc:290] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.551970 2604 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "43bec2f03fc849099e254f6335e9d532" instance_seqno: 1761335273959853) as {username='slave'} at 127.0.252.193:55789; Asking this server to re-register.
I20251024 19:48:20.552440 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c92b080805845c6af8f1cb160110915" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:20.553117 1586 leader_election.cc:343] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.553604 1767 tablet_bootstrap.cc:492] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:20.553857 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 935) took 3003 ms. Trace:
I20251024 19:48:20.553941 1776 rpcz_store.cc:276] 1024 19:48:17.549939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.549995 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:20.490503 (+2940508us) ts_tablet_manager.cc:1151] Deleting tablet 2602335327fc489fa154056c5ef5138f
1024 19:48:20.490515 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.520055 (+ 29540us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.536979 (+ 16924us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.553837 (+ 16858us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2940347}
I20251024 19:48:20.554289 1776 tablet_replica.cc:333] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:20.554345 1768 tablet_bootstrap.cc:654] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.554533 1776 raft_consensus.cc:2243] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:20.554898 1776 raft_consensus.cc:2272] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.555457 2604 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "10a9536207e343f188e0ef709081d04e" instance_seqno: 1761335274554669) as {username='slave'} at 127.0.252.194:35781; Asking this server to re-register.
I20251024 19:48:20.555789 1776 ts_tablet_manager.cc:1916] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.557081 1418 heartbeater.cc:461] Registering TS with master...
W20251024 19:48:20.446513 1444 consensus_peers.cc:597] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b240965605904533bd2af99f61aaef26. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:20.557745 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 2602335327fc489fa154056c5ef5138f (table default.Table_059135c028074f7fb4bc26da343a4de1 [id=d35a25cbeefd47d39a5cba0bed321da4]) successfully deleted
W20251024 19:48:20.557857 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 111369us
I20251024 19:48:20.491006 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet a3259823b622461fa823841925b3d203 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.559389 1557 heartbeater.cc:461] Registering TS with master...
I20251024 19:48:20.559787 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 904) took 3412 ms. Trace:
I20251024 19:48:20.559892 1775 rpcz_store.cc:276] 1024 19:48:17.147385 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.148658 (+ 1273us) service_pool.cc:225] Handling call
1024 19:48:20.559479 (+3410821us) ts_tablet_manager.cc:1151] Deleting tablet 4243f11bf2b34b30b10766c19e2c6409
1024 19:48:20.559493 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.559767 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3409699}
I20251024 19:48:20.560391 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 2602335327fc489fa154056c5ef5138f (table default.Table_059135c028074f7fb4bc26da343a4de1 [id=d35a25cbeefd47d39a5cba0bed321da4]) successfully deleted
W20251024 19:48:20.560738 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4243f11bf2b34b30b10766c19e2c6409 because the tablet was not found. No further retry: Not found: Tablet not found: 4243f11bf2b34b30b10766c19e2c6409
I20251024 19:48:20.496537 1766 tablet_bootstrap.cc:492] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:20.514648 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 38818c7997c84cedad5f9b09eef77105 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.562130 1766 ts_tablet_manager.cc:1403] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.129s user 0.003s sys 0.003s
I20251024 19:48:20.562917 1766 raft_consensus.cc:359] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.563081 1766 raft_consensus.cc:385] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.563128 1766 raft_consensus.cc:740] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.563328 1766 consensus_queue.cc:260] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.563936 1766 ts_tablet_manager.cc:1434] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:20.534807 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet a2cc3f3d02324c1e867b8fc159ad6d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.550807 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c92b080805845c6af8f1cb160110915" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:20.565440 1505 raft_consensus.cc:2468] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:20.566028 1586 leader_election.cc:304] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:20.566578 3141 raft_consensus.cc:2804] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:20.566655 3141 raft_consensus.cc:493] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:20.566713 3141 raft_consensus.cc:3060] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.567133 1131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef8906492b1a4368936fac95c11e1f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 932 } } ignore_live_leader: false dest_uuid: "82e82847f8904b59b03f446b7fce6233" is_pre_election: true
I20251024 19:48:20.570292 3141 raft_consensus.cc:515] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.572177 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c92b080805845c6af8f1cb160110915" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:20.572383 1505 raft_consensus.cc:3060] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.575464 1505 raft_consensus.cc:2468] T 1c92b080805845c6af8f1cb160110915 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:20.576154 1586 leader_election.cc:304] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:20.576493 3137 raft_consensus.cc:2804] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:20.576633 3137 raft_consensus.cc:697] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:20.576964 3137 consensus_queue.cc:237] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.577430 3141 leader_election.cc:290] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.577452 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c92b080805845c6af8f1cb160110915" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:20.578241 1586 leader_election.cc:343] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.578809 2604 ts_manager.cc:194] Registered new tserver with Master: af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:20.580688 2603 ts_manager.cc:194] Registered new tserver with Master: 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:20.583825 2604 ts_manager.cc:194] Registered new tserver with Master: 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.590526 1774 ts_tablet_manager.cc:1929] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.591468 1766 tablet_bootstrap.cc:492] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:20.593415 1766 tablet_bootstrap.cc:654] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.595913 1775 tablet_replica.cc:333] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:20.596415 1775 raft_consensus.cc:2243] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:20.596915 1767 tablet_bootstrap.cc:654] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.597306 1775 raft_consensus.cc:2272] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.598745 1775 ts_tablet_manager.cc:1916] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.603127 1767 tablet_bootstrap.cc:492] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:20.603909 1767 ts_tablet_manager.cc:1403] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.000s
I20251024 19:48:20.605043 1767 raft_consensus.cc:359] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.608404 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c80fa7789cc34b718c6de9760db48a17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:20.608628 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4ecf5be7f6504214bc074337a9ddd17a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:20.609025 1767 raft_consensus.cc:385] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.609105 1767 raft_consensus.cc:740] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
W20251024 19:48:20.609135 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c80fa7789cc34b718c6de9760db48a17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.609349 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3e12a4891d024f54bb915e9f4dd0c60f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:20.609555 1767 consensus_queue.cc:260] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:20.609951 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e12a4891d024f54bb915e9f4dd0c60f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.610672 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9aa983432fce4338b09f4182277c9825" } modify_external_catalogs: true
I20251024 19:48:20.610944 1767 ts_tablet_manager.cc:1434] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251024 19:48:20.610999 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9aa983432fce4338b09f4182277c9825" } modify_external_catalogs: true
I20251024 19:48:20.612046 1767 tablet_bootstrap.cc:492] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:20.613299 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_941306808730402d8ac4181a7a4877f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.613905 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_941306808730402d8ac4181a7a4877f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.614670 3163 raft_consensus.cc:493] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.614815 3163 raft_consensus.cc:515] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.615497 3163 leader_election.cc:290] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:20.618942 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ecf5be7f6504214bc074337a9ddd17a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.619232 1202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef8906492b1a4368936fac95c11e1f73" candidate_term: 2 candidate_status { last_received { term: 1 index: 932 } } ignore_live_leader: false dest_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" is_pre_election: true
I20251024 19:48:20.600385 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 250f106664be4b5aaa330dbb4e43cb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.616508 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee68243b4a664575b679950603bb250e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:20.619931 3163 consensus_queue.cc:1048] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:20.620110 1505 raft_consensus.cc:2468] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:20.620565 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6811639fccc645fd9ccd5bfe038fbadd" } new_table_name: "default.Table_df4b29031df04d61ba08e9741f230fda" modify_external_catalogs: true
I20251024 19:48:20.623138 1586 leader_election.cc:304] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:20.623409 2923 raft_consensus.cc:2804] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:20.623489 2923 raft_consensus.cc:493] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:20.623549 2923 raft_consensus.cc:3060] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.625545 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dcb0f784408346bda089b634ed26412e" } modify_external_catalogs: true
I20251024 19:48:20.625784 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dcb0f784408346bda089b634ed26412e" } modify_external_catalogs: true
I20251024 19:48:20.626618 2923 raft_consensus.cc:515] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.627368 2923 leader_election.cc:290] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.627831 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee68243b4a664575b679950603bb250e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:20.627974 1766 tablet_bootstrap.cc:492] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:20.628011 1511 raft_consensus.cc:3060] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.628118 1766 ts_tablet_manager.cc:1403] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.003s
I20251024 19:48:20.628142 1774 log.cc:1199] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/16bc737104ab46b3b071a0760dbc847d
I20251024 19:48:20.628772 1774 ts_tablet_manager.cc:1950] T 16bc737104ab46b3b071a0760dbc847d P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:20.629222 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 872) took 2520 ms. Trace:
I20251024 19:48:20.629310 1774 rpcz_store.cc:276] 1024 19:48:18.109164 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.110094 (+ 930us) service_pool.cc:225] Handling call
1024 19:48:20.449083 (+2338989us) ts_tablet_manager.cc:1151] Deleting tablet 16bc737104ab46b3b071a0760dbc847d
1024 19:48:20.449092 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.562688 (+113596us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.590405 (+ 27717us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.629198 (+ 38793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2239239}
I20251024 19:48:20.629503 1766 raft_consensus.cc:359] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.629720 1766 raft_consensus.cc:385] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.629786 1766 raft_consensus.cc:740] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.630040 1766 consensus_queue.cc:260] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.630113 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 16bc737104ab46b3b071a0760dbc847d (table default.Table_775e17f58fef4f2b850035a03e411a08 [id=6239ceae506442b79bbe79c48813b49e]) successfully deleted
I20251024 19:48:20.630537 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 876) took 2350 ms. Trace:
I20251024 19:48:20.630810 1766 ts_tablet_manager.cc:1434] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251024 19:48:20.631073 1766 tablet_bootstrap.cc:492] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:20.631711 1511 raft_consensus.cc:2468] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:20.632326 1586 leader_election.cc:304] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:20.630610 1774 rpcz_store.cc:276] 1024 19:48:18.279815 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.286487 (+ 6672us) service_pool.cc:225] Handling call
1024 19:48:20.630388 (+2343901us) ts_tablet_manager.cc:1151] Deleting tablet 7a65fc59de5346d6a9344c5310455e53
1024 19:48:20.630398 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.630516 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2343615}
I20251024 19:48:20.590685 1768 tablet_bootstrap.cc:492] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:20.616663 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee68243b4a664575b679950603bb250e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:20.627926 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee68243b4a664575b679950603bb250e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:20.633101 1768 ts_tablet_manager.cc:1403] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.134s user 0.002s sys 0.002s
W20251024 19:48:20.633322 1586 leader_election.cc:343] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.633661 2923 raft_consensus.cc:2804] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
W20251024 19:48:20.633705 1586 leader_election.cc:343] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.633898 2923 raft_consensus.cc:697] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:20.634176 2923 consensus_queue.cc:237] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:20.634352 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7a65fc59de5346d6a9344c5310455e53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a65fc59de5346d6a9344c5310455e53 already in progress: creating tablet
I20251024 19:48:20.634608 1768 raft_consensus.cc:359] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.634899 1768 raft_consensus.cc:385] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.634707 2590 leader_election.cc:304] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef8906492b1a4368936fac95c11e1f73; no voters: 82e82847f8904b59b03f446b7fce6233, dc8f71a5b70749a4a3e3a599bcb3d22c
I20251024 19:48:20.635007 1768 raft_consensus.cc:740] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.635385 3115 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:48:20.636057 1768 consensus_queue.cc:260] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.639053 1768 ts_tablet_manager.cc:1434] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251024 19:48:20.640015 1768 tablet_bootstrap.cc:492] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:20.653838 1767 tablet_bootstrap.cc:654] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.656085 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2def15e9c34c4a1ca47b1600a926f7c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.656741 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2def15e9c34c4a1ca47b1600a926f7c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.658537 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 94fc8c5716af47faa4e831f05fce8610 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:17 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.659364 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet 3a99e9b3fe044c7eb14fa9bffa903873 from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:20.659624 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 300952047a324a25a7c687eff20d6e56 consensus state has no leader
I20251024 19:48:20.662237 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 875) took 2387 ms. Trace:
I20251024 19:48:20.662372 1774 rpcz_store.cc:276] 1024 19:48:18.274664 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.288128 (+ 13464us) service_pool.cc:225] Handling call
1024 19:48:20.662082 (+2373954us) ts_tablet_manager.cc:1151] Deleting tablet 81e3e2f8082d4c229e617f34f5eb5a78
1024 19:48:20.662096 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.662213 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2373736}
I20251024 19:48:20.662820 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 877) took 2369 ms. Trace:
I20251024 19:48:20.662922 1774 rpcz_store.cc:276] 1024 19:48:18.293079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.307653 (+ 14574us) service_pool.cc:225] Handling call
1024 19:48:20.662716 (+2355063us) ts_tablet_manager.cc:1151] Deleting tablet f16ef86c5eb24489a66d6cd4bfa804fa
1024 19:48:20.662725 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.662805 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2354871}
I20251024 19:48:20.663192 1774 tablet_replica.cc:333] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:20.663381 1774 raft_consensus.cc:2243] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.663591 1774 raft_consensus.cc:2272] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.664206 1774 ts_tablet_manager.cc:1916] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:20.669073 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 81e3e2f8082d4c229e617f34f5eb5a78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81e3e2f8082d4c229e617f34f5eb5a78 already in progress: creating tablet
W20251024 19:48:20.669395 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet f16ef86c5eb24489a66d6cd4bfa804fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f16ef86c5eb24489a66d6cd4bfa804fa already in progress: creating tablet
I20251024 19:48:20.684103 1768 tablet_bootstrap.cc:654] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.684113 2857 raft_consensus.cc:493] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.684280 2857 raft_consensus.cc:515] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.684914 2857 leader_election.cc:290] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.685567 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38818c7997c84cedad5f9b09eef77105" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:20.686285 1775 ts_tablet_manager.cc:1929] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.687177 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38818c7997c84cedad5f9b09eef77105" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:20.689924 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_bc3b64103c5b40db85bb106ea58c5c62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.690689 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc3b64103c5b40db85bb106ea58c5c62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:20.692653 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d5aeb6bab67b4c1ea34aa869731de12a" } new_table_name: "default.Table_0a3260e932f447998a886cb5e967fd46" modify_external_catalogs: true
W20251024 19:48:20.686371 1444 leader_election.cc:343] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:20.693315 1447 leader_election.cc:343] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.693837 1447 leader_election.cc:304] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:20.694067 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2134712eb3c54d96879eef43e8fb1d2a" } new_table_name: "default.Table_8523c29419f0445aa4da2ffdc6480f1a" modify_external_catalogs: true
I20251024 19:48:20.694455 2857 raft_consensus.cc:2749] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:20.695533 1767 tablet_bootstrap.cc:492] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:20.695746 1767 ts_tablet_manager.cc:1403] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.085s user 0.004s sys 0.000s
I20251024 19:48:20.696199 1775 log.cc:1199] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/5d129bdc876e425aabcf1c99997e319a
I20251024 19:48:20.696899 1767 raft_consensus.cc:359] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.697131 1767 raft_consensus.cc:385] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.697230 1767 raft_consensus.cc:740] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.697520 1767 consensus_queue.cc:260] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.699316 1766 tablet_bootstrap.cc:654] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.700572 1767 ts_tablet_manager.cc:1434] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:48:20.700984 1767 tablet_bootstrap.cc:492] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:20.702502 1767 tablet_bootstrap.cc:654] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.706336 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4bd52fc39eef4537bafea34b959fa79c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.707257 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bd52fc39eef4537bafea34b959fa79c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:20.711282 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2671108031db449cae2135835fc5f739 consensus state has no leader
I20251024 19:48:20.711761 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_58d5b87034ec4242b1ab84dabf05f01b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.712425 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58d5b87034ec4242b1ab84dabf05f01b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:20.714028 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b2992a738b440d798f5e61cc45ec088 consensus state has no leader
I20251024 19:48:20.714510 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet 30bce965315d43948ecb8282c829c557 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:20.717999 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e7975a91f4674c4fa7904e3b7bc780dc" } new_table_name: "default.Table_34b29b41568a41e8a7cd22e7ad65949f" modify_external_catalogs: true
I20251024 19:48:20.719849 3240 catalog_manager.cc:1269] Loaded cluster ID: 57054fc705ad43a6b8ba5fe9c17ff20a
I20251024 19:48:20.719944 3240 catalog_manager.cc:1562] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: loading cluster ID for follower catalog manager: success
I20251024 19:48:20.696647 1775 ts_tablet_manager.cc:1950] T 5d129bdc876e425aabcf1c99997e319a P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:20.720979 1774 ts_tablet_manager.cc:1929] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.721891 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 911) took 3571 ms. Trace:
I20251024 19:48:20.722015 1774 log.cc:1199] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/f83b9290f00145a780318a43249eec12
I20251024 19:48:20.722525 1774 ts_tablet_manager.cc:1950] T f83b9290f00145a780318a43249eec12 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:20.722590 1775 rpcz_store.cc:276] 1024 19:48:17.150280 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.150337 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:20.560268 (+3409931us) ts_tablet_manager.cc:1151] Deleting tablet 5d129bdc876e425aabcf1c99997e319a
1024 19:48:20.569665 (+ 9397us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.679104 (+109439us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.686173 (+ 7069us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.721871 (+ 35698us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3409702}
I20251024 19:48:20.723269 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 879) took 2334 ms. Trace:
I20251024 19:48:20.723304 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 5d129bdc876e425aabcf1c99997e319a (table default.Table_a4b63032d2224a6d95765cb93137f4a2 [id=d28fd80220c64f8496a09f5d4e712874]) successfully deleted
I20251024 19:48:20.723716 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 912) took 3572 ms. Trace:
I20251024 19:48:20.723449 1774 rpcz_store.cc:276] 1024 19:48:18.388844 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.389158 (+ 314us) service_pool.cc:225] Handling call
1024 19:48:20.663151 (+2273993us) ts_tablet_manager.cc:1151] Deleting tablet f83b9290f00145a780318a43249eec12
1024 19:48:20.663159 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.702342 (+ 39183us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.705461 (+ 3119us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.723252 (+ 17791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2273814}
I20251024 19:48:20.724082 1775 rpcz_store.cc:276] 1024 19:48:17.150781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.150832 (+ 51us) service_pool.cc:225] Handling call
1024 19:48:20.723612 (+3572780us) ts_tablet_manager.cc:1151] Deleting tablet b88a6a5237d146f5b6f082ca3af25b2b
1024 19:48:20.723621 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.723700 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3572563}
I20251024 19:48:20.724310 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet f83b9290f00145a780318a43249eec12 (table default.Table_420ebc050cf440e9a59a18e26f7b83af [id=be1cc9d047554ea6a9afa531fb08a81d]) successfully deleted
I20251024 19:48:20.724610 1774 tablet_replica.cc:333] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
W20251024 19:48:20.724766 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b88a6a5237d146f5b6f082ca3af25b2b because the tablet was not found. No further retry: Not found: Tablet not found: b88a6a5237d146f5b6f082ca3af25b2b
I20251024 19:48:20.725042 1775 tablet_replica.cc:333] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:20.725031 1774 raft_consensus.cc:2243] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.725203 1774 raft_consensus.cc:2272] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.725452 1775 raft_consensus.cc:2243] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.725926 1775 raft_consensus.cc:2272] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.726661 1775 ts_tablet_manager.cc:1916] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.730556 1774 ts_tablet_manager.cc:1916] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.732734 1776 ts_tablet_manager.cc:1929] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.732851 1776 log.cc:1199] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/62667092d7e244cca1020ec220ea9124
I20251024 19:48:20.733177 1776 ts_tablet_manager.cc:1950] T 62667092d7e244cca1020ec220ea9124 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:20.734700 1767 tablet_bootstrap.cc:492] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:20.734843 1767 ts_tablet_manager.cc:1403] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20251024 19:48:20.735837 1767 raft_consensus.cc:359] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.736016 1767 raft_consensus.cc:385] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.736083 1767 raft_consensus.cc:740] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.736326 1767 consensus_queue.cc:260] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.742997 1775 ts_tablet_manager.cc:1929] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251024 19:48:20.743463 1177 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2671108031db449cae2135835fc5f739 (table default.Table_0a3260e932f447998a886cb5e967fd46 [id=2f6b7b685af84337bd64141ef2a9e992], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2671108031db449cae2135835fc5f739 consensus state has no leader
I20251024 19:48:20.744010 1767 ts_tablet_manager.cc:1434] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251024 19:48:20.744318 1767 tablet_bootstrap.cc:492] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:20.745724 1767 tablet_bootstrap.cc:654] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.745849 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f2cdf925c04543daa2bdb70bd9f67dbe" } modify_external_catalogs: true
I20251024 19:48:20.746054 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f2cdf925c04543daa2bdb70bd9f67dbe" } modify_external_catalogs: true
I20251024 19:48:20.746752 3208 raft_consensus.cc:493] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.746896 3208 raft_consensus.cc:515] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.747545 3208 leader_election.cc:290] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:20.747504 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9bcef35dbbd649ccbf09b1067e78bcc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:20.748046 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bcef35dbbd649ccbf09b1067e78bcc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:20.749950 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91bb076b4b0d4b04955043248460d04c consensus state has no leader
I20251024 19:48:20.750383 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_90159463cd7741cba6ccb29461b104cd" } new_table_name: "default.Table_b057911324a844ac83247dcfdf437dcc" modify_external_catalogs: true
I20251024 19:48:20.744812 2857 raft_consensus.cc:493] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.750849 2857 raft_consensus.cc:515] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.751478 2857 leader_election.cc:290] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.752676 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0266cef4e2034303978d7456c47ea98c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:20.753233 1447 leader_election.cc:343] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.745379 1766 tablet_bootstrap.cc:492] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:20.758332 1766 ts_tablet_manager.cc:1403] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.127s user 0.005s sys 0.005s
I20251024 19:48:20.759150 1766 raft_consensus.cc:359] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.759327 1766 raft_consensus.cc:385] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.759378 1766 raft_consensus.cc:740] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.759567 1766 consensus_queue.cc:260] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.760056 1766 ts_tablet_manager.cc:1434] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:20.760370 1766 tablet_bootstrap.cc:492] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:20.749970 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94fc8c5716af47faa4e831f05fce8610" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:20.763334 1766 tablet_bootstrap.cc:654] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.765033 1775 log.cc:1199] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/8a4cb86bc5e745d5b4ee932f0a38b3c7
I20251024 19:48:20.765686 1775 ts_tablet_manager.cc:1950] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:20.768059 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 909) took 3618 ms. Trace:
I20251024 19:48:20.748271 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94fc8c5716af47faa4e831f05fce8610" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:20.752337 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0266cef4e2034303978d7456c47ea98c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:20.770393 2923 raft_consensus.cc:493] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251024 19:48:20.770483 1444 leader_election.cc:343] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.770524 2923 raft_consensus.cc:515] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:20.770586 1444 leader_election.cc:304] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:20.770566 1305 leader_election.cc:304] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:20.770931 2857 raft_consensus.cc:2749] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:20.770978 3208 raft_consensus.cc:2749] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:20.770057 1775 rpcz_store.cc:276] 1024 19:48:17.149659 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.151267 (+ 1608us) service_pool.cc:225] Handling call
1024 19:48:20.725000 (+3573733us) ts_tablet_manager.cc:1151] Deleting tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7
1024 19:48:20.725009 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.739587 (+ 14578us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.742887 (+ 3300us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.768031 (+ 25144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3573511}
I20251024 19:48:20.771842 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a1031d785b646b095ece9be666d2a77" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:20.771891 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 (table default.Table_5006c140e0964b569953c4f03495c697 [id=338beb9c6956484daef970b28d5e4b06]) successfully deleted
I20251024 19:48:20.771998 1505 raft_consensus.cc:2468] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:20.772344 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a1031d785b646b095ece9be666d2a77" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:20.772635 1586 leader_election.cc:304] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:20.773010 3137 raft_consensus.cc:2804] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251024 19:48:20.773052 1586 leader_election.cc:343] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.773087 3137 raft_consensus.cc:493] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:20.773147 3137 raft_consensus.cc:3060] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.773427 2923 leader_election.cc:290] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:20.769685 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 936) took 3212 ms. Trace:
I20251024 19:48:20.773794 1775 tablet_replica.cc:333] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:20.773706 1776 rpcz_store.cc:276] 1024 19:48:17.556859 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.556913 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:20.554240 (+2997327us) ts_tablet_manager.cc:1151] Deleting tablet 62667092d7e244cca1020ec220ea9124
1024 19:48:20.554252 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.559275 (+ 5023us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.732565 (+173290us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.769644 (+ 37079us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2997162}
I20251024 19:48:20.773981 1775 raft_consensus.cc:2243] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.774163 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 62667092d7e244cca1020ec220ea9124 (table default.Table_06b07944861041f8a66001a6555d7659 [id=96c4981300dc48e6bf55fc633164ed63]) successfully deleted
I20251024 19:48:20.776050 3137 raft_consensus.cc:515] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:20.776773 3137 leader_election.cc:290] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:20.777194 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a1031d785b646b095ece9be666d2a77" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:20.777359 1505 raft_consensus.cc:3060] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.780120 1505 raft_consensus.cc:2468] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:20.780836 1586 leader_election.cc:304] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:20.781334 3137 raft_consensus.cc:2804] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:20.781526 3137 raft_consensus.cc:697] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:20.781543 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a1031d785b646b095ece9be666d2a77" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:20.781852 3137 consensus_queue.cc:237] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:20.782368 1586 leader_election.cc:343] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.783635 1776 tablet_replica.cc:333] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:20.786623 1774 ts_tablet_manager.cc:1929] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.786886 1776 raft_consensus.cc:2243] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:20.787684 1776 raft_consensus.cc:2272] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.789631 1767 tablet_bootstrap.cc:492] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:20.789767 1767 ts_tablet_manager.cc:1403] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20251024 19:48:20.790400 1767 raft_consensus.cc:359] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.790555 1767 raft_consensus.cc:385] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.790596 1767 raft_consensus.cc:740] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.790944 1767 consensus_queue.cc:260] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.791450 1767 ts_tablet_manager.cc:1434] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:20.791766 1767 tablet_bootstrap.cc:492] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:20.794065 1776 ts_tablet_manager.cc:1916] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.798615 1774 log.cc:1199] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/4193750331874d7ba9bbe6279c3a8e37
I20251024 19:48:20.799362 1774 ts_tablet_manager.cc:1950] T 4193750331874d7ba9bbe6279c3a8e37 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:20.800168 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 880) took 2411 ms. Trace:
I20251024 19:48:20.800258 1767 tablet_bootstrap.cc:654] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.800733 1774 rpcz_store.cc:276] 1024 19:48:18.388958 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.389862 (+ 904us) service_pool.cc:225] Handling call
1024 19:48:20.724568 (+2334706us) ts_tablet_manager.cc:1151] Deleting tablet 4193750331874d7ba9bbe6279c3a8e37
1024 19:48:20.724577 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.737933 (+ 13356us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.786503 (+ 48570us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.800147 (+ 13644us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2334527}
I20251024 19:48:20.801681 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 4193750331874d7ba9bbe6279c3a8e37 (table default.Table_18cdd8018c304deba24dd3429fdb48fc [id=3172f9d08e8649c289a02ad1cd61177d]) successfully deleted
I20251024 19:48:20.802367 1774 tablet_replica.cc:333] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:20.802838 1774 raft_consensus.cc:2243] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.807310 1774 raft_consensus.cc:2272] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.808861 1774 ts_tablet_manager.cc:1916] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.814533 3137 raft_consensus.cc:493] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.814808 3137 raft_consensus.cc:515] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.814945 1766 tablet_bootstrap.cc:492] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:20.815083 1766 ts_tablet_manager.cc:1403] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.055s user 0.014s sys 0.002s
I20251024 19:48:20.815788 3137 leader_election.cc:290] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:20.816306 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d840888183ec4ef7bb245363c5bac322" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:20.816851 1586 leader_election.cc:343] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.817066 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d840888183ec4ef7bb245363c5bac322" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:20.817265 1505 raft_consensus.cc:2468] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:20.817731 1586 leader_election.cc:304] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:20.817992 3137 raft_consensus.cc:2804] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:20.818059 3137 raft_consensus.cc:493] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:20.818131 3137 raft_consensus.cc:3060] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.805148 1768 tablet_bootstrap.cc:492] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:20.818413 1768 ts_tablet_manager.cc:1403] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.178s user 0.004s sys 0.000s
I20251024 19:48:20.819607 1768 raft_consensus.cc:359] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.819797 1768 raft_consensus.cc:385] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.819864 1768 raft_consensus.cc:740] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.820071 1768 consensus_queue.cc:260] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.820693 1768 ts_tablet_manager.cc:1434] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:20.833459 2923 consensus_queue.cc:1048] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:20.840930 3137 raft_consensus.cc:515] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.842278 3137 leader_election.cc:290] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:20.843900 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d840888183ec4ef7bb245363c5bac322" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:20.844118 1511 raft_consensus.cc:3060] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.846000 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d840888183ec4ef7bb245363c5bac322" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:20.846606 1364 raft_consensus.cc:3060] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.848234 1511 raft_consensus.cc:2468] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:20.848284 1768 tablet_bootstrap.cc:492] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:20.848704 1586 leader_election.cc:304] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:20.849012 2923 raft_consensus.cc:2804] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:20.849110 2923 raft_consensus.cc:697] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:20.849349 2923 consensus_queue.cc:237] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.849265 1766 raft_consensus.cc:359] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.849543 1766 raft_consensus.cc:385] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.849742 1766 raft_consensus.cc:740] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.850049 1766 consensus_queue.cc:260] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.850786 1766 ts_tablet_manager.cc:1434] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.036s user 0.000s sys 0.002s
W20251024 19:48:20.850960 1586 leader_election.cc:343] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.851428 1766 tablet_bootstrap.cc:492] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:20.854480 1766 tablet_bootstrap.cc:654] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.858230 1767 tablet_bootstrap.cc:492] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:20.858487 1767 ts_tablet_manager.cc:1403] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.007s
I20251024 19:48:20.858592 1774 ts_tablet_manager.cc:1929] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.858712 1774 log.cc:1199] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/2602335327fc489fa154056c5ef5138f
I20251024 19:48:20.859143 1774 ts_tablet_manager.cc:1950] T 2602335327fc489fa154056c5ef5138f P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:20.859478 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 881) took 2470 ms. Trace:
I20251024 19:48:20.859552 1774 rpcz_store.cc:276] 1024 19:48:18.389088 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.390057 (+ 969us) service_pool.cc:225] Handling call
1024 19:48:20.802310 (+2412253us) ts_tablet_manager.cc:1151] Deleting tablet 2602335327fc489fa154056c5ef5138f
1024 19:48:20.802322 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.826294 (+ 23972us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.858461 (+ 32167us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.859461 (+ 1000us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2412102}
I20251024 19:48:20.860013 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 887) took 2447 ms. Trace:
I20251024 19:48:20.860080 1774 rpcz_store.cc:276] 1024 19:48:18.412720 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.412764 (+ 44us) service_pool.cc:225] Handling call
1024 19:48:20.859890 (+2447126us) ts_tablet_manager.cc:1151] Deleting tablet b95989b98c77472faa7bf940cfa60432
1024 19:48:20.859899 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.859997 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2446973}
I20251024 19:48:20.860404 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 883) took 2448 ms. Trace:
I20251024 19:48:20.860476 1774 rpcz_store.cc:276] 1024 19:48:18.411666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.411730 (+ 64us) service_pool.cc:225] Handling call
1024 19:48:20.860301 (+2448571us) ts_tablet_manager.cc:1151] Deleting tablet 36ccb0cfba7a4a14b36d38897f4a9ef0
1024 19:48:20.860311 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.860386 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2413003}
I20251024 19:48:20.860755 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 902) took 1967 ms. Trace:
I20251024 19:48:20.860828 1774 rpcz_store.cc:276] 1024 19:48:18.893505 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.893563 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:20.860648 (+1967085us) ts_tablet_manager.cc:1151] Deleting tablet 54bc7c7764584e11948d2831a4e5e0b6
1024 19:48:20.860655 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.860741 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1966876}
I20251024 19:48:20.861116 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 903) took 1966 ms. Trace:
I20251024 19:48:20.861191 1774 rpcz_store.cc:276] 1024 19:48:18.894737 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.894834 (+ 97us) service_pool.cc:225] Handling call
1024 19:48:20.861027 (+1966193us) ts_tablet_manager.cc:1151] Deleting tablet de2ebb79d4fe4a1bbac372bb26f370ed
1024 19:48:20.861035 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.861103 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1966042}
I20251024 19:48:20.861496 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 904) took 1958 ms. Trace:
I20251024 19:48:20.861469 1767 raft_consensus.cc:359] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.862010 1767 raft_consensus.cc:385] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.862109 1767 raft_consensus.cc:740] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.861943 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 2602335327fc489fa154056c5ef5138f (table default.Table_059135c028074f7fb4bc26da343a4de1 [id=d35a25cbeefd47d39a5cba0bed321da4]) successfully deleted
W20251024 19:48:20.862658 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b95989b98c77472faa7bf940cfa60432 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b95989b98c77472faa7bf940cfa60432 already in progress: creating tablet
I20251024 19:48:20.862882 1767 consensus_queue.cc:260] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:20.863524 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 54bc7c7764584e11948d2831a4e5e0b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54bc7c7764584e11948d2831a4e5e0b6 already in progress: creating tablet
I20251024 19:48:20.863726 1767 ts_tablet_manager.cc:1434] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20251024 19:48:20.863970 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet de2ebb79d4fe4a1bbac372bb26f370ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de2ebb79d4fe4a1bbac372bb26f370ed already in progress: creating tablet
I20251024 19:48:20.864029 1767 tablet_bootstrap.cc:492] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:20.864497 1775 raft_consensus.cc:2272] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.866070 1775 ts_tablet_manager.cc:1916] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.876729 3240 catalog_manager.cc:1584] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: acquiring CA information for follower catalog manager: success
W20251024 19:48:20.876699 1586 consensus_peers.cc:597] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:20.877279 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04e13290af90466d8766f571f72f20a4 has no consensus state
I20251024 19:48:20.873750 2857 raft_consensus.cc:493] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.881722 2857 raft_consensus.cc:515] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.882480 2857 leader_election.cc:290] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.861565 1774 rpcz_store.cc:276] 1024 19:48:18.903293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.903347 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:20.861404 (+1958057us) ts_tablet_manager.cc:1151] Deleting tablet c9761b71a7294c78bccb5be86529dbe8
1024 19:48:20.861411 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.861481 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1957867}
I20251024 19:48:20.882995 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e7b3288182744969e217d4f174fe4ea" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:20.883723 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d6c796b5abfe4ba0b02c8b2e41101f45" } modify_external_catalogs: true
I20251024 19:48:20.884011 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d6c796b5abfe4ba0b02c8b2e41101f45" } modify_external_catalogs: true
I20251024 19:48:20.884547 1774 tablet_replica.cc:333] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:20.884763 1774 raft_consensus.cc:2243] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.884928 1774 raft_consensus.cc:2272] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.885489 1774 ts_tablet_manager.cc:1916] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:20.887356 1586 consensus_peers.cc:597] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:20.888443 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e7b3288182744969e217d4f174fe4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:20.889513 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet c9761b71a7294c78bccb5be86529dbe8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9761b71a7294c78bccb5be86529dbe8 already in progress: creating tablet
W20251024 19:48:20.889707 1444 leader_election.cc:343] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.891446 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e7b3288182744969e217d4f174fe4ea" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:20.892087 1447 leader_election.cc:343] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.892215 1447 leader_election.cc:304] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:20.892477 2857 raft_consensus.cc:2749] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:20.900332 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet cb0405765c6146cbbb187af96a60be2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:20.901548 1767 tablet_bootstrap.cc:654] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.905377 1776 ts_tablet_manager.cc:1929] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.907595 1768 tablet_bootstrap.cc:654] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.254145 2660 rpcz_store.cc:276] 1024 19:48:12.660142 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:12.660309 (+ 167us) service_pool.cc:225] Handling call
1024 19:48:17.574118 (+4913809us) raft_consensus.cc:1471] Updating replica for 338 ops
1024 19:48:17.662728 (+ 88610us) raft_consensus.cc:1534] Early marking committed up to index 594
1024 19:48:17.850710 (+187982us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:17.850720 (+ 10us) write_op.cc:454] Released schema lock
1024 19:48:17.882945 (+ 32225us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:17.882955 (+ 10us) write_op.cc:454] Released schema lock
1024 19:48:17.951511 (+ 68556us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:17.951516 (+ 5us) write_op.cc:454] Released schema lock
1024 19:48:17.970648 (+ 19132us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:17.970660 (+ 12us) write_op.cc:454] Released schema lock
1024 19:48:17.976709 (+ 6049us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:17.976715 (+ 6us) write_op.cc:454] Released schema lock
1024 19:48:17.983523 (+ 6808us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:17.983529 (+ 6us) write_op.cc:454] Released schema lock
1024 19:48:17.985059 (+ 1530us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:17.985066 (+ 7us) write_op.cc:454] Released schema lock
1024 19:48:18.044456 (+ 59390us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.044468 (+ 12us) write_op.cc:454] Released schema lock
1024 19:48:18.057033 (+ 12565us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.057039 (+ 6us) write_op.cc:454] Released schema lock
1024 19:48:18.058673 (+ 1634us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.058679 (+ 6us) write_op.cc:454] Released schema lock
1024 19:48:18.059390 (+ 711us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.059395 (+ 5us) write_op.cc:454] Released schema lock
1024 19:48:18.075516 (+ 16121us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.075524 (+ 8us) write_op.cc:454] Released schema lock
1024 19:48:18.076639 (+ 1115us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.076644 (+ 5us) write_op.cc:454] Released schema lock
1024 19:48:18.080285 (+ 3641us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.080291 (+ 6us) write_op.cc:454] Released schema lock
1024 19:48:18.081022 (+ 731us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.081027 (+ 5us) write_op.cc:454] Released schema lock
1024 19:48:18.082319 (+ 1292us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.082324 (+ 5us) write_op.cc:454] Released schema lock
1024 19:48:18.083688 (+ 1364us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.083694 (+ 6us) write_op.cc:454] Released schema lock
1024 19:48:18.084358 (+ 664us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.084363 (+ 5us) write_op.cc:454] Released schema lock
1024 19:48:18.085005 (+ 642us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.085009 (+ 4us) write_op.cc:454] Released schema lock
1024 19:48:18.085684 (+ 675us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.085688 (+ 4us) write_op.cc:454] Released schema lock
1024 19:48:18.122880 (+ 37192us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.122888 (+ 8us) write_op.cc:454] Released schema lock
1024 19:48:18.131289 (+ 8401us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.131296 (+ 7us) write_op.cc:454] Released schema lock
1024 19:48:18.135468 (+ 4172us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.135475 (+ 7us) write_op.cc:454] Released schema lock
1024 19:48:18.136644 (+ 1169us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:18.136649 (+ 5us) write_op.cc:454] Released schema lock
1024 19:48:18.137237 (+ 588us) raft_consensus.cc:1539] Triggering prepare for 338 ops
1024 19:48:18.137641 (+ 404us) write_op.cc:270] Start()
1024 19:48:18.137688 (+ 47us) write_op.cc:276] Timestamp: P: 1761335285749046 usec, L: 0
1024 19:48:18.137994 (+ 306us) write_op.cc:270] Start()
1024 19:48:18.138024 (+ 30us) write_op.cc:276] Timestamp: P: 1761335285782069 usec, L: 0
1024 19:48:18.138183 (+ 159us) write_op.cc:270] Start()
1024 19:48:18.138208 (+ 25us) write_op.cc:276] Timestamp: P: 1761335285806543 usec, L: 0
1024 19:48:18.138363 (+ 155us) write_op.cc:270] Start()
1024 19:48:18.138384 (+ 21us) write_op.cc:276] Timestamp: P: 1761335285811878 usec, L: 0
1024 19:48:18.138629 (+ 245us) write_op.cc:270] Start()
1024 19:48:18.138652 (+ 23us) write_op.cc:276] Timestamp: P: 1761335285812543 usec, L: 0
1024 19:48:18.138792 (+ 140us) write_op.cc:270] Start()
1024 19:48:18.138829 (+ 37us) write_op.cc:276] Timestamp: P: 1761335285813138 usec, L: 0
1024 19:48:18.138995 (+ 166us) write_op.cc:270] Start()
1024 19:48:18.139018 (+ 23us) write_op.cc:276] Timestamp: P: 1761335285827876 usec, L: 0
1024 19:48:18.139186 (+ 168us) write_op.cc:270] Start()
1024 19:48:18.139207 (+ 21us) write_op.cc:276] Timestamp: P: 1761335285828536 usec, L: 0
1024 19:48:18.139358 (+ 151us) write_op.cc:270] Start()
1024 19:48:18.139382 (+ 24us) write_op.cc:276] Timestamp: P: 1761335285829119 usec, L: 0
1024 19:48:18.139538 (+ 156us) write_op.cc:270] Start()
1024 19:48:18.139558 (+ 20us) write_op.cc:276] Timestamp: P: 1761335285829769 usec, L: 0
1024 19:48:18.139684 (+ 126us) write_op.cc:270] Start()
1024 19:48:18.139705 (+ 21us) write_op.cc:276] Timestamp: P: 1761335285834425 usec, L: 0
1024 19:48:18.139837 (+ 132us) write_op.cc:270] Start()
1024 19:48:18.139856 (+ 19us) write_op.cc:276] Timestamp: P: 1761335285834812 usec, L: 0
1024 19:48:18.140000 (+ 144us) write_op.cc:270] Start()
1024 19:48:18.140020 (+ 20us) write_op.cc:276] Timestamp: P: 1761335285902438 usec, L: 0
1024 19:48:18.140155 (+ 135us) write_op.cc:270] Start()
1024 19:48:18.140176 (+ 21us) write_op.cc:276] Timestamp: P: 1761335285951518 usec, L: 0
1024 19:48:18.140332 (+ 156us) write_op.cc:270] Start()
1024 19:48:18.140356 (+ 24us) write_op.cc:276] Timestamp: P: 1761335285957527 usec, L: 0
1024 19:48:18.140499 (+ 143us) write_op.cc:270] Start()
1024 19:48:18.140519 (+ 20us) write_op.cc:276] Timestamp: P: 1761335285958111 usec, L: 0
1024 19:48:18.140676 (+ 157us) write_op.cc:270] Start()
1024 19:48:18.140697 (+ 21us) write_op.cc:276] Timestamp: P: 1761335286069202 usec, L: 0
1024 19:48:18.140858 (+ 161us) write_op.cc:270] Start()
1024 19:48:18.140886 (+ 28us) write_op.cc:276] Timestamp: P: 1761335286071657 usec, L: 0
1024 19:48:18.141065 (+ 179us) write_op.cc:270] Start()
1024 19:48:18.141083 (+ 18us) write_op.cc:276] Timestamp: P: 1761335286112594 usec, L: 0
1024 19:48:18.141211 (+ 128us) write_op.cc:270] Start()
1024 19:48:18.141230 (+ 19us) write_op.cc:276] Timestamp: P: 1761335286148002 usec, L: 0
1024 19:48:18.141414 (+ 184us) write_op.cc:270] Start()
1024 19:48:18.141438 (+ 24us) write_op.cc:276] Timestamp: P: 1761335286151606 usec, L: 0
1024 19:48:18.308041 (+166603us) write_op.cc:270] Start()
1024 19:48:18.308139 (+ 98us) write_op.cc:276] Timestamp: P: 1761335286156284 usec, L: 0
1024 19:48:18.308363 (+ 224us) write_op.cc:270] Start()
1024 19:48:18.308388 (+ 25us) write_op.cc:276] Timestamp: P: 1761335286184154 usec, L: 0
1024 19:48:18.308564 (+ 176us) write_op.cc:270] Start()
1024 19:48:18.308588 (+ 24us) write_op.cc:276] Timestamp: P: 1761335286193134 usec, L: 0
1024 19:48:18.308814 (+ 226us) write_op.cc:270] Start()
1024 19:48:18.308836 (+ 22us) write_op.cc:276] Timestamp: P: 1761335286212638 usec, L: 0
1024 19:48:18.308983 (+ 147us) write_op.cc:270] Start()
1024 19:48:18.309003 (+ 20us) write_op.cc:276] Timestamp: P: 1761335286219652 usec, L: 0
1024 19:48:18.309140 (+ 137us) write_op.cc:270] Start()
1024 19:48:18.309159 (+ 19us) write_op.cc:276] Timestamp: P: 1761335286220195 usec, L: 0
1024 19:48:18.309285 (+ 126us) write_op.cc:270] Start()
1024 19:48:18.309302 (+ 17us) write_op.cc:276] Timestamp: P: 1761335286225136 usec, L: 0
1024 19:48:18.309410 (+ 108us) write_op.cc:270] Start()
1024 19:48:18.309426 (+ 16us) write_op.cc:276] Timestamp: P: 1761335286261618 usec, L: 0
1024 19:48:18.309536 (+ 110us) write_op.cc:270] Start()
1024 19:48:18.309552 (+ 16us) write_op.cc:276] Timestamp: P: 1761335286313818 usec, L: 0
1024 19:48:18.309721 (+ 169us) write_op.cc:270] Start()
1024 19:48:18.309743 (+ 22us) write_op.cc:276] Timestamp: P: 1761335286316605 usec, L: 0
1024 19:48:18.309885 (+ 142us) write_op.cc:270] Start()
1024 19:48:18.309907 (+ 22us) write_op.cc:276] Timestamp: P: 1761335286317413 usec, L: 0
1024 19:48:18.310064 (+ 157us) write_op.cc:270] Start()
1024 19:48:18.310085 (+ 21us) write_op.cc:276] Timestamp: P: 1761335286318214 usec, L: 0
1024 19:48:18.310302 (+ 217us) write_op.cc:270] Start()
1024 19:48:18.310323 (+ 21us) write_op.cc:276] Timestamp: P: 1761335286318966 usec, L: 0
1024 19:48:18.310469 (+ 146us) write_op.cc:270] Start()
1024 19:48:18.310501 (+ 32us) write_op.cc:276] Timestamp: P: 1761335286319619 usec, L: 0
1024 19:48:18.310645 (+ 144us) write_op.cc:270] Start()
1024 19:48:18.310666 (+ 21us) write_op.cc:276] Timestamp: P: 1761335286320207 usec, L: 0
1024 19:48:18.310814 (+ 148us) write_op.cc:270] Start()
1024 19:48:18.310835 (+ 21us) write_op.cc:276] Timestamp: P: 1761335286320778 usec, L: 0
1024 19:48:18.310965 (+ 130us) write_op.cc:270] Start()
1024 19:48:18.310982 (+ 17us) write_op.cc:276] Timestamp: P: 1761335286376981 usec, L: 0
1024 19:48:18.311113 (+ 131us) write_op.cc:270] Start()
1024 19:48:18.311136 (+ 23us) write_op.cc:276] Timestamp: P: 1761335286433907 usec, L: 0
1024 19:48:18.311279 (+ 143us) write_op.cc:270] Start()
1024 19:48:18.311299 (+ 20us) write_op.cc:276] Timestamp: P: 1761335286436805 usec, L: 0
1024 19:48:18.311435 (+ 136us) write_op.cc:270] Start()
1024 19:48:18.311457 (+ 22us) write_op.cc:276] Timestamp: P: 1761335286437528 usec, L: 0
1024 19:48:18.311632 (+ 175us) write_op.cc:270] Start()
1024 19:48:18.311653 (+ 21us) write_op.cc:276] Timestamp: P: 1761335286442299 usec, L: 0
1024 19:48:18.353361 (+ 41708us) write_op.cc:270] Start()
1024 19:48:18.353437 (+ 76us) write_op.cc:276] Timestamp: P: 1761335286452763 usec, L: 0
1024 19:48:18.354056 (+ 619us) write_op.cc:270] Start()
1024 19:48:18.354087 (+ 31us) write_op.cc:276] Timestamp: P: 1761335286458503 usec, L: 0
1024 19:48:18.354674 (+ 587us) write_op.cc:270] Start()
1024 19:48:18.354733 (+ 59us) write_op.cc:276] Timestamp: P: 1761335286460244 usec, L: 0
1024 19:48:18.355016 (+ 283us) write_op.cc:270] Start()
1024 19:48:18.355074 (+ 58us) write_op.cc:276] Timestamp: P: 1761335286468916 usec, L: 0
1024 19:48:18.355449 (+ 375us) write_op.cc:270] Start()
1024 19:48:18.355478 (+ 29us) write_op.cc:276] Timestamp: P: 1761335286472431 usec, L: 0
1024 19:48:18.355851 (+ 373us) write_op.cc:270] Start()
1024 19:48:18.355879 (+ 28us) write_op.cc:276] Timestamp: P: 1761335286615318 usec, L: 0
1024 19:48:18.356404 (+ 525us) write_op.cc:270] Start()
1024 19:48:18.356494 (+ 90us) write_op.cc:276] Timestamp: P: 1761335286656577 usec, L: 0
1024 19:48:18.357079 (+ 585us) write_op.cc:270] Start()
1024 19:48:18.357132 (+ 53us) write_op.cc:276] Timestamp: P: 1761335286672583 usec, L: 0
1024 19:48:18.357453 (+ 321us) write_op.cc:270] Start()
1024 19:48:18.357506 (+ 53us) write_op.cc:276] Timestamp: P: 1761335286719886 usec, L: 0
1024 19:48:18.514843 (+157337us) write_op.cc:270] Start()
1024 19:48:18.514884 (+ 41us) write_op.cc:276] Timestamp: P: 1761335286720798 usec, L: 0
1024 19:48:18.515026 (+ 142us) write_op.cc:270] Start()
1024 19:48:18.515046 (+ 20us) write_op.cc:276] Timestamp: P: 1761335286721459 usec, L: 0
1024 19:48:18.515198 (+ 152us) write_op.cc:270] Start()
1024 19:48:18.515216 (+ 18us) write_op.cc:276] Timestamp: P: 1761335286722067 usec, L: 0
1024 19:48:18.515336 (+ 120us) write_op.cc:270] Start()
1024 19:48:18.515352 (+ 16us) write_op.cc:276] Timestamp: P: 1761335286722742 usec, L: 0
1024 19:48:18.515460 (+ 108us) write_op.cc:270] Start()
1024 19:48:18.515478 (+ 18us) write_op.cc:276] Timestamp: P: 1761335286723448 usec, L: 0
1024 19:48:18.515609 (+ 131us) write_op.cc:270] Start()
1024 19:48:18.515626 (+ 17us) write_op.cc:276] Timestamp: P: 1761335286748175 usec, L: 0
1024 19:48:18.515768 (+ 142us) write_op.cc:270] Start()
1024 19:48:18.515789 (+ 21us) write_op.cc:276] Timestamp: P: 1761335286754835 usec, L: 0
1024 19:48:18.515944 (+ 155us) write_op.cc:270] Start()
1024 19:48:18.515965 (+ 21us) write_op.cc:276] Timestamp: P: 1761335286757890 usec, L: 0
1024 19:48:18.516137 (+ 172us) write_op.cc:270] Start()
1024 19:48:18.516156 (+ 19us) write_op.cc:276] Timestamp: P: 1761335286820967 usec, L: 0
1024 19:48:18.516296 (+ 140us) write_op.cc:270] Start()
1024 19:48:18.516316 (+ 20us) write_op.cc:276] Timestamp: P: 1761335286867612 usec, L: 0
1024 19:48:18.516455 (+ 139us) write_op.cc:270] Start()
1024 19:48:18.516474 (+ 19us) write_op.cc:276] Timestamp: P: 1761335286885736 usec, L: 0
1024 19:48:18.516613 (+ 139us) write_op.cc:270] Start()
1024 19:48:18.516632 (+ 19us) write_op.cc:276] Timestamp: P: 1761335286887573 usec, L: 0
1024 19:48:18.516817 (+ 185us) write_op.cc:270] Start()
1024 19:48:18.516842 (+ 25us) write_op.cc:276] Timestamp: P: 1761335286888796 usec, L: 0
1024 19:48:18.517025 (+ 183us) write_op.cc:270] Start()
1024 19:48:18.517045 (+ 20us) write_op.cc:276] Timestamp: P: 1761335286890009 usec, L: 0
1024 19:48:18.517189 (+ 144us) write_op.cc:270] Start()
1024 19:48:18.517209 (+ 20us) write_op.cc:276] Timestamp: P: 1761335286920792 usec, L: 0
1024 19:48:18.517362 (+ 153us) write_op.cc:270] Start()
1024 19:48:18.517394 (+ 32us) write_op.cc:276] Timestamp: P: 1761335287197299 usec, L: 0
1024 19:48:18.517538 (+ 144us) write_op.cc:270] Start()
1024 19:48:18.517557 (+ 19us) write_op.cc:276] Timestamp: P: 1761335287197938 usec, L: 0
1024 19:48:18.729173 (+211616us) write_op.cc:270] Start()
1024 19:48:18.729239 (+ 66us) write_op.cc:276] Timestamp: P: 1761335287207559 usec, L: 0
1024 19:48:18.729382 (+ 143us) write_op.cc:270] Start()
1024 19:48:18.729403 (+ 21us) write_op.cc:276] Timestamp: P: 1761335287231917 usec, L: 0
1024 19:48:18.729537 (+ 134us) write_op.cc:270] Start()
1024 19:48:18.729555 (+ 18us) write_op.cc:276] Timestamp: P: 1761335287235421 usec, L: 0
1024 19:48:18.729723 (+ 168us) write_op.cc:270] Start()
1024 19:48:18.729743 (+ 20us) write_op.cc:276] Timestamp: P: 1761335287236039 usec, L: 0
1024 19:48:18.729872 (+ 129us) write_op.cc:270] Start()
1024 19:48:18.729890 (+ 18us) write_op.cc:276] Timestamp: P: 1761335287241852 usec, L: 0
1024 19:48:18.730151 (+ 261us) write_op.cc:270] Start()
1024 19:48:18.730182 (+ 31us) write_op.cc:276] Timestamp: P: 1761335287302541 usec, L: 0
1024 19:48:18.730356 (+ 174us) write_op.cc:270] Start()
1024 19:48:18.730378 (+ 22us) write_op.cc:276] Timestamp: P: 1761335287314539 usec, L: 0
1024 19:48:18.730533 (+ 155us) write_op.cc:270] Start()
1024 19:48:18.730560 (+ 27us) write_op.cc:276] Timestamp: P: 1761335287338451 usec, L: 0
1024 19:48:18.730719 (+ 159us) write_op.cc:270] Start()
1024 19:48:18.730754 (+ 35us) write_op.cc:276] Timestamp: P: 1761335287349394 usec, L: 0
1024 19:48:18.730892 (+ 138us) write_op.cc:270] Start()
1024 19:48:18.730913 (+ 21us) write_op.cc:276] Timestamp: P: 1761335287361162 usec, L: 0
1024 19:48:18.731058 (+ 145us) write_op.cc:270] Start()
1024 19:48:18.731151 (+ 93us) write_op.cc:276] Timestamp: P: 1761335287377632 usec, L: 0
1024 19:48:18.731456 (+ 305us) write_op.cc:270] Start()
1024 19:48:18.731480 (+ 24us) write_op.cc:276] Timestamp: P: 1761335287381396 usec, L: 0
1024 19:48:18.731650 (+ 170us) write_op.cc:270] Start()
1024 19:48:18.731671 (+ 21us) write_op.cc:276] Timestamp: P: 1761335287383831 usec, L: 0
1024 19:48:18.731843 (+ 172us) write_op.cc:270] Start()
1024 19:48:18.731863 (+ 20us) write_op.cc:276] Timestamp: P: 1761335287384812 usec, L: 0
1024 19:48:18.731994 (+ 131us) write_op.cc:270] Start()
1024 19:48:18.732014 (+ 20us) write_op.cc:276] Timestamp: P: 1761335287398408 usec, L: 0
1024 19:48:18.732160 (+ 146us) write_op.cc:270] Start()
1024 19:48:18.732178 (+ 18us) write_op.cc:276] Timestamp: P: 1761335287446253 usec, L: 0
1024 19:48:18.732321 (+ 143us) write_op.cc:270] Start()
1024 19:48:18.732338 (+ 17us) write_op.cc:276] Timestamp: P: 1761335287468205 usec, L: 0
1024 19:48:18.732441 (+ 103us) write_op.cc:270] Start()
1024 19:48:18.732456 (+ 15us) write_op.cc:276] Timestamp: P: 1761335287471580 usec, L: 0
1024 19:48:18.732569 (+ 113us) write_op.cc:270] Start()
1024 19:48:18.732585 (+ 16us) write_op.cc:276] Timestamp: P: 1761335287516262 usec, L: 0
1024 19:48:18.882709 (+150124us) write_op.cc:270] Start()
1024 19:48:18.882760 (+ 51us) write_op.cc:276] Timestamp: P: 1761335287523097 usec, L: 0
1024 19:48:18.883085 (+ 325us) write_op.cc:270] Start()
1024 19:48:18.883111 (+ 26us) write_op.cc:276] Timestamp: P: 1761335287525323 usec, L: 0
1024 19:48:18.883275 (+ 164us) write_op.cc:270] Start()
1024 19:48:18.883294 (+ 19us) write_op.cc:276] Timestamp: P: 1761335287534022 usec, L: 0
1024 19:48:18.883524 (+ 230us) write_op.cc:270] Start()
1024 19:48:18.883552 (+ 28us) write_op.cc:276] Timestamp: P: 1761335287541831 usec, L: 0
1024 19:48:18.883753 (+ 201us) write_op.cc:270] Start()
1024 19:48:18.883775 (+ 22us) write_op.cc:276] Timestamp: P: 1761335287605531 usec, L: 0
1024 19:48:18.883987 (+ 212us) write_op.cc:270] Start()
1024 19:48:18.884016 (+ 29us) write_op.cc:276] Timestamp: P: 1761335287645388 usec, L: 0
1024 19:48:18.884241 (+ 225us) write_op.cc:270] Start()
1024 19:48:18.884261 (+ 20us) write_op.cc:276] Timestamp: P: 1761335287658849 usec, L: 0
1024 19:48:18.884478 (+ 217us) write_op.cc:270] Start()
1024 19:48:18.884503 (+ 25us) write_op.cc:276] Timestamp: P: 1761335287664122 usec, L: 0
1024 19:48:18.884773 (+ 270us) write_op.cc:270] Start()
1024 19:48:18.884801 (+ 28us) write_op.cc:276] Timestamp: P: 1761335287688282 usec, L: 0
1024 19:48:18.885006 (+ 205us) write_op.cc:270] Start()
1024 19:48:18.885025 (+ 19us) write_op.cc:276] Timestamp: P: 1761335287709349 usec, L: 0
1024 19:48:18.885202 (+ 177us) write_op.cc:270] Start()
1024 19:48:18.885228 (+ 26us) write_op.cc:276] Timestamp: P: 1761335287736617 usec, L: 0
1024 19:48:18.885401 (+ 173us) write_op.cc:270] Start()
1024 19:48:18.885419 (+ 18us) write_op.cc:276] Timestamp: P: 1761335287747741 usec, L: 0
1024 19:48:18.885588 (+ 169us) write_op.cc:270] Start()
1024 19:48:18.885614 (+ 26us) write_op.cc:276] Timestamp: P: 1761335287883087 usec, L: 0
1024 19:48:18.885836 (+ 222us) write_op.cc:270] Start()
1024 19:48:18.885858 (+ 22us) write_op.cc:276] Timestamp: P: 1761335287935698 usec, L: 0
1024 19:48:18.886058 (+ 200us) write_op.cc:270] Start()
1024 19:48:18.886089 (+ 31us) write_op.cc:276] Timestamp: P: 1761335287957050 usec, L: 0
1024 19:48:18.886290 (+ 201us) write_op.cc:270] Start()
1024 19:48:18.886343 (+ 53us) write_op.cc:276] Timestamp: P: 1761335287990902 usec, L: 0
1024 19:48:18.886578 (+ 235us) write_op.cc:270] Start()
1024 19:48:18.886604 (+ 26us) write_op.cc:276] Timestamp: P: 1761335288022103 usec, L: 0
1024 19:48:18.886874 (+ 270us) write_op.cc:270] Start()
1024 19:48:18.886906 (+ 32us) write_op.cc:276] Timestamp: P: 1761335288037600 usec, L: 0
1024 19:48:18.887138 (+ 232us) write_op.cc:270] Start()
1024 19:48:18.887156 (+ 18us) write_op.cc:276] Timestamp: P: 1761335288076041 usec, L: 0
1024 19:48:18.887342 (+ 186us) write_op.cc:270] Start()
1024 19:48:18.887364 (+ 22us) write_op.cc:276] Timestamp: P: 1761335288091196 usec, L: 0
1024 19:48:18.887573 (+ 209us) write_op.cc:270] Start()
1024 19:48:18.887595 (+ 22us) write_op.cc:276] Timestamp: P: 1761335288102820 usec, L: 0
1024 19:48:18.887726 (+ 131us) write_op.cc:270] Start()
1024 19:48:18.887744 (+ 18us) write_op.cc:276] Timestamp: P: 1761335288124205 usec, L: 0
1024 19:48:18.958249 (+ 70505us) write_op.cc:270] Start()
1024 19:48:18.958299 (+ 50us) write_op.cc:276] Timestamp: P: 1761335288136332 usec, L: 0
1024 19:48:18.959007 (+ 708us) write_op.cc:270] Start()
1024 19:48:18.959035 (+ 28us) write_op.cc:276] Timestamp: P: 1761335288146929 usec, L: 0
1024 19:48:18.959426 (+ 391us) write_op.cc:270] Start()
1024 19:48:18.959454 (+ 28us) write_op.cc:276] Timestamp: P: 1761335288174941 usec, L: 0
1024 19:48:18.959623 (+ 169us) write_op.cc:270] Start()
1024 19:48:18.959644 (+ 21us) write_op.cc:276] Timestamp: P: 1761335288187639 usec, L: 0
1024 19:48:18.959774 (+ 130us) write_op.cc:270] Start()
1024 19:48:18.959792 (+ 18us) write_op.cc:276] Timestamp: P: 1761335288201373 usec, L: 0
1024 19:48:18.959938 (+ 146us) write_op.cc:270] Start()
1024 19:48:18.959970 (+ 32us) write_op.cc:276] Timestamp: P: 1761335288206092 usec, L: 0
1024 19:48:18.960173 (+ 203us) write_op.cc:270] Start()
1024 19:48:18.960203 (+ 30us) write_op.cc:276] Timestamp: P: 1761335288225286 usec, L: 0
1024 19:48:18.960360 (+ 157us) write_op.cc:270] Start()
1024 19:48:18.960383 (+ 23us) write_op.cc:276] Timestamp: P: 1761335288234193 usec, L: 0
1024 19:48:18.960571 (+ 188us) write_op.cc:270] Start()
1024 19:48:18.960588 (+ 17us) write_op.cc:276] Timestamp: P: 1761335288305820 usec, L: 0
1024 19:48:18.960733 (+ 145us) write_op.cc:270] Start()
1024 19:48:18.960763 (+ 30us) write_op.cc:276] Timestamp: P: 1761335288337088 usec, L: 0
1024 19:48:18.960891 (+ 128us) write_op.cc:270] Start()
1024 19:48:18.960910 (+ 19us) write_op.cc:276] Timestamp: P: 1761335288370092 usec, L: 0
1024 19:48:18.961154 (+ 244us) write_op.cc:270] Start()
1024 19:48:18.961184 (+ 30us) write_op.cc:276] Timestamp: P: 1761335288388802 usec, L: 0
1024 19:48:18.961362 (+ 178us) write_op.cc:270] Start()
1024 19:48:18.961386 (+ 24us) write_op.cc:276] Timestamp: P: 1761335288419820 usec, L: 0
1024 19:48:18.961564 (+ 178us) write_op.cc:270] Start()
1024 19:48:19.054795 (+ 93231us) write_op.cc:276] Timestamp: P: 1761335288420425 usec, L: 0
1024 19:48:19.055186 (+ 391us) write_op.cc:270] Start()
1024 19:48:19.055218 (+ 32us) write_op.cc:276] Timestamp: P: 1761335288420978 usec, L: 0
1024 19:48:19.055398 (+ 180us) write_op.cc:270] Start()
1024 19:48:19.055431 (+ 33us) write_op.cc:276] Timestamp: P: 1761335288458056 usec, L: 0
1024 19:48:19.055582 (+ 151us) write_op.cc:270] Start()
1024 19:48:19.055604 (+ 22us) write_op.cc:276] Timestamp: P: 1761335288501031 usec, L: 0
1024 19:48:19.055801 (+ 197us) write_op.cc:270] Start()
1024 19:48:19.055825 (+ 24us) write_op.cc:276] Timestamp: P: 1761335288501807 usec, L: 0
1024 19:48:19.056062 (+ 237us) write_op.cc:270] Start()
1024 19:48:19.056087 (+ 25us) write_op.cc:276] Timestamp: P: 1761335288502418 usec, L: 0
1024 19:48:19.056340 (+ 253us) write_op.cc:270] Start()
1024 19:48:19.056367 (+ 27us) write_op.cc:276] Timestamp: P: 1761335288538653 usec, L: 0
1024 19:48:19.056681 (+ 314us) write_op.cc:270] Start()
1024 19:48:19.056709 (+ 28us) write_op.cc:276] Timestamp: P: 1761335288755276 usec, L: 0
1024 19:48:19.056888 (+ 179us) write_op.cc:270] Start()
1024 19:48:19.056916 (+ 28us) write_op.cc:276] Timestamp: P: 1761335288756029 usec, L: 0
1024 19:48:19.057067 (+ 151us) write_op.cc:270] Start()
1024 19:48:19.057090 (+ 23us) write_op.cc:276] Timestamp: P: 1761335288756753 usec, L: 0
1024 19:48:19.057236 (+ 146us) write_op.cc:270] Start()
1024 19:48:19.057265 (+ 29us) write_op.cc:276] Timestamp: P: 1761335288757425 usec, L: 0
1024 19:48:19.057442 (+ 177us) write_op.cc:270] Start()
1024 19:48:19.057465 (+ 23us) write_op.cc:276] Timestamp: P: 1761335288758821 usec, L: 0
1024 19:48:19.057653 (+ 188us) write_op.cc:270] Start()
1024 19:48:19.057670 (+ 17us) write_op.cc:276] Timestamp: P: 1761335288760314 usec, L: 0
1024 19:48:19.057802 (+ 132us) write_op.cc:270] Start()
1024 19:48:19.057826 (+ 24us) write_op.cc:276] Timestamp: P: 1761335288760959 usec, L: 0
1024 19:48:19.057976 (+ 150us) write_op.cc:270] Start()
1024 19:48:19.057998 (+ 22us) write_op.cc:276] Timestamp: P: 1761335288766729 usec, L: 0
1024 19:48:19.058157 (+ 159us) write_op.cc:270] Start()
1024 19:48:19.058180 (+ 23us) write_op.cc:276] Timestamp: P: 1761335288767271 usec, L: 0
1024 19:48:19.058337 (+ 157us) write_op.cc:270] Start()
1024 19:48:19.058360 (+ 23us) write_op.cc:276] Timestamp: P: 1761335288770334 usec, L: 0
1024 19:48:19.058547 (+ 187us) write_op.cc:270] Start()
1024 19:48:19.153654 (+ 95107us) write_op.cc:276] Timestamp: P: 1761335288816140 usec, L: 0
1024 19:48:19.153956 (+ 302us) write_op.cc:270] Start()
1024 19:48:19.153994 (+ 38us) write_op.cc:276] Timestamp: P: 1761335288817144 usec, L: 0
1024 19:48:19.154121 (+ 127us) write_op.cc:270] Start()
1024 19:48:19.154138 (+ 17us) write_op.cc:276] Timestamp: P: 1761335288818983 usec, L: 0
1024 19:48:19.154255 (+ 117us) write_op.cc:270] Start()
1024 19:48:19.154275 (+ 20us) write_op.cc:276] Timestamp: P: 1761335288819632 usec, L: 0
1024 19:48:19.154434 (+ 159us) write_op.cc:270] Start()
1024 19:48:19.154461 (+ 27us) write_op.cc:276] Timestamp: P: 1761335288847108 usec, L: 0
1024 19:48:19.154613 (+ 152us) write_op.cc:270] Start()
1024 19:48:19.154633 (+ 20us) write_op.cc:276] Timestamp: P: 1761335288929113 usec, L: 0
1024 19:48:19.154771 (+ 138us) write_op.cc:270] Start()
1024 19:48:19.154788 (+ 17us) write_op.cc:276] Timestamp: P: 1761335288929958 usec, L: 0
1024 19:48:19.154893 (+ 105us) write_op.cc:270] Start()
1024 19:48:19.154909 (+ 16us) write_op.cc:276] Timestamp: P: 1761335288930667 usec, L: 0
1024 19:48:19.155021 (+ 112us) write_op.cc:270] Start()
1024 19:48:19.155037 (+ 16us) write_op.cc:276] Timestamp: P: 1761335288933045 usec, L: 0
1024 19:48:19.155146 (+ 109us) write_op.cc:270] Start()
1024 19:48:19.155160 (+ 14us) write_op.cc:276] Timestamp: P: 1761335288963606 usec, L: 0
1024 19:48:19.155293 (+ 133us) write_op.cc:270] Start()
1024 19:48:19.155317 (+ 24us) write_op.cc:276] Timestamp: P: 1761335288972336 usec, L: 0
1024 19:48:19.155481 (+ 164us) write_op.cc:270] Start()
1024 19:48:19.155504 (+ 23us) write_op.cc:276] Timestamp: P: 1761335288974854 usec, L: 0
1024 19:48:19.155762 (+ 258us) write_op.cc:270] Start()
1024 19:48:19.155789 (+ 27us) write_op.cc:276] Timestamp: P: 1761335289006314 usec, L: 0
1024 19:48:19.155985 (+ 196us) write_op.cc:270] Start()
1024 19:48:19.156006 (+ 21us) write_op.cc:276] Timestamp: P: 1761335289013246 usec, L: 0
1024 19:48:19.156208 (+ 202us) write_op.cc:270] Start()
1024 19:48:19.156231 (+ 23us) write_op.cc:276] Timestamp: P: 1761335289015054 usec, L: 0
1024 19:48:19.158217 (+ 1986us) write_op.cc:270] Start()
1024 19:48:19.158246 (+ 29us) write_op.cc:276] Timestamp: P: 1761335289017494 usec, L: 0
1024 19:48:19.158556 (+ 310us) write_op.cc:270] Start()
1024 19:48:19.158581 (+ 25us) write_op.cc:276] Timestamp: P: 1761335289019775 usec, L: 0
1024 19:48:19.158743 (+ 162us) write_op.cc:270] Start()
1024 19:48:19.158765 (+ 22us) write_op.cc:276] Timestamp: P: 1761335289028778 usec, L: 0
1024 19:48:19.158986 (+ 221us) write_op.cc:270] Start()
1024 19:48:19.159011 (+ 25us) write_op.cc:276] Timestamp: P: 1761335289050576 usec, L: 0
1024 19:48:19.159203 (+ 192us) write_op.cc:270] Start()
1024 19:48:19.159224 (+ 21us) write_op.cc:276] Timestamp: P: 1761335289052101 usec, L: 0
1024 19:48:19.159457 (+ 233us) write_op.cc:270] Start()
1024 19:48:19.159482 (+ 25us) write_op.cc:276] Timestamp: P: 1761335289057307 usec, L: 0
1024 19:48:19.159701 (+ 219us) write_op.cc:270] Start()
1024 19:48:19.159724 (+ 23us) write_op.cc:276] Timestamp: P: 1761335289213197 usec, L: 0
1024 19:48:19.159885 (+ 161us) write_op.cc:270] Start()
1024 19:48:19.159903 (+ 18us) write_op.cc:276] Timestamp: P: 1761335289213906 usec, L: 0
1024 19:48:19.160110 (+ 207us) write_op.cc:270] Start()
1024 19:48:19.160132 (+ 22us) write_op.cc:276] Timestamp: P: 1761335289214491 usec, L: 0
1024 19:48:19.160261 (+ 129us) write_op.cc:270] Start()
1024 19:48:19.160279 (+ 18us) write_op.cc:276] Timestamp: P: 1761335289240135 usec, L: 0
1024 19:48:19.160425 (+ 146us) write_op.cc:270] Start()
1024 19:48:19.160443 (+ 18us) write_op.cc:276] Timestamp: P: 1761335289244724 usec, L: 0
1024 19:48:19.160555 (+ 112us) write_op.cc:270] Start()
1024 19:48:19.160571 (+ 16us) write_op.cc:276] Timestamp: P: 1761335289246666 usec, L: 0
1024 19:48:19.160760 (+ 189us) write_op.cc:270] Start()
1024 19:48:19.160854 (+ 94us) write_op.cc:276] Timestamp: P: 1761335289250360 usec, L: 0
1024 19:48:19.161100 (+ 246us) write_op.cc:270] Start()
1024 19:48:19.161120 (+ 20us) write_op.cc:276] Timestamp: P: 1761335289251586 usec, L: 0
1024 19:48:19.161314 (+ 194us) write_op.cc:270] Start()
1024 19:48:19.161336 (+ 22us) write_op.cc:276] Timestamp: P: 1761335289252440 usec, L: 0
1024 19:48:19.161497 (+ 161us) write_op.cc:270] Start()
1024 19:48:19.161632 (+ 135us) write_op.cc:276] Timestamp: P: 1761335289253329 usec, L: 0
1024 19:48:19.161842 (+ 210us) write_op.cc:270] Start()
1024 19:48:19.161862 (+ 20us) write_op.cc:276] Timesta
I20251024 19:48:20.926949 1766 tablet_bootstrap.cc:492] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:20.927212 1766 ts_tablet_manager.cc:1403] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.004s
I20251024 19:48:20.927326 1775 ts_tablet_manager.cc:1929] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.929102 1775 log.cc:1199] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/4193750331874d7ba9bbe6279c3a8e37
I20251024 19:48:20.929464 1775 ts_tablet_manager.cc:1950] T 4193750331874d7ba9bbe6279c3a8e37 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:20.929643 1774 ts_tablet_manager.cc:1929] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.929831 1774 log.cc:1199] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/9489a01434a24d9ca94791252950725c
I20251024 19:48:20.930289 1774 ts_tablet_manager.cc:1950] T 9489a01434a24d9ca94791252950725c P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:20.930330 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 910) took 3780 ms. Trace:
I20251024 19:48:20.930413 1775 rpcz_store.cc:276] 1024 19:48:17.149980 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.150039 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:20.772346 (+3622307us) ts_tablet_manager.cc:1151] Deleting tablet 4193750331874d7ba9bbe6279c3a8e37
1024 19:48:20.773750 (+ 1404us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.910202 (+136452us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.927220 (+ 17018us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.930307 (+ 3087us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3620037}
I20251024 19:48:20.930797 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 905) took 1712 ms. Trace:
I20251024 19:48:20.930931 1774 rpcz_store.cc:276] 1024 19:48:19.218733 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.218789 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:20.884493 (+1665704us) ts_tablet_manager.cc:1151] Deleting tablet 9489a01434a24d9ca94791252950725c
1024 19:48:20.884505 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.919593 (+ 35088us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.929430 (+ 9837us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.930780 (+ 1350us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1665496}
I20251024 19:48:20.931412 1774 tablet_replica.cc:333] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:20.928068 1766 raft_consensus.cc:359] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.931563 1766 raft_consensus.cc:385] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.931631 1766 raft_consensus.cc:740] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.931876 1766 consensus_queue.cc:260] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.937829 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 9489a01434a24d9ca94791252950725c (table default.Table_1c89d244042a4852a9cb4113f6726332 [id=ba207dd9990d4a0e85eaefbb4908dd02]) successfully deleted
I20251024 19:48:20.938084 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 4193750331874d7ba9bbe6279c3a8e37 (table default.Table_18cdd8018c304deba24dd3429fdb48fc [id=3172f9d08e8649c289a02ad1cd61177d]) successfully deleted
I20251024 19:48:20.939517 2701 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:48:20.939682 2701 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: This master's current role is: LEADER
I20251024 19:48:20.941704 1766 ts_tablet_manager.cc:1434] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251024 19:48:20.942023 1766 tablet_bootstrap.cc:492] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:20.942696 1768 tablet_bootstrap.cc:492] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:20.942829 1768 ts_tablet_manager.cc:1403] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.095s user 0.001s sys 0.004s
I20251024 19:48:20.943375 1766 tablet_bootstrap.cc:654] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.943879 1768 raft_consensus.cc:359] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.944031 1768 raft_consensus.cc:385] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.944082 1768 raft_consensus.cc:740] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.944427 1768 consensus_queue.cc:260] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.944972 1768 ts_tablet_manager.cc:1434] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251024 19:48:20.945243 1768 tablet_bootstrap.cc:492] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:20.946929 1768 tablet_bootstrap.cc:654] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:20.951241 2857 raft_consensus.cc:493] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.951295 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 913) took 3800 ms. Trace:
I20251024 19:48:20.951375 1775 rpcz_store.cc:276] 1024 19:48:17.151320 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.152293 (+ 973us) service_pool.cc:225] Handling call
1024 19:48:20.951194 (+3798901us) ts_tablet_manager.cc:1151] Deleting tablet cef80e4d14bb4e719bceebf43041d079
1024 19:48:20.951204 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.951279 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3798742}
I20251024 19:48:20.951378 2857 raft_consensus.cc:515] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:20.951674 1775 tablet_replica.cc:333] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:20.951871 1775 raft_consensus.cc:2243] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.952013 1775 raft_consensus.cc:2272] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.952133 2857 leader_election.cc:290] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.952523 1775 ts_tablet_manager.cc:1916] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.953521 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b632acc1348e49f8a1eea019c007780c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:20.911753 3137 raft_consensus.cc:493] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.953866 3137 raft_consensus.cc:515] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:20.954296 1447 leader_election.cc:343] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.954497 3137 leader_election.cc:290] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:20.954713 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet cef80e4d14bb4e719bceebf43041d079 because the tablet was not found. No further retry: Not found: Tablet not found: cef80e4d14bb4e719bceebf43041d079
I20251024 19:48:20.955153 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3d80885fcc642b8a75da30cd4abecbd" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:20.955479 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3d80885fcc642b8a75da30cd4abecbd" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:20.955652 1505 raft_consensus.cc:2468] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
W20251024 19:48:20.955636 1586 leader_election.cc:343] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.928972 1767 tablet_bootstrap.cc:492] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:20.955991 1586 leader_election.cc:304] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:20.956092 1767 ts_tablet_manager.cc:1403] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.092s user 0.001s sys 0.011s
I20251024 19:48:20.956249 3137 raft_consensus.cc:2804] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:20.956305 3137 raft_consensus.cc:493] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:20.956359 3137 raft_consensus.cc:3060] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.952771 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b632acc1348e49f8a1eea019c007780c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:20.956841 1444 leader_election.cc:343] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.956933 1444 leader_election.cc:304] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:20.957456 2857 raft_consensus.cc:2749] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:20.959102 3137 raft_consensus.cc:515] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.959697 3137 leader_election.cc:290] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:20.960543 1767 raft_consensus.cc:359] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.960697 1767 raft_consensus.cc:385] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:20.960785 1767 raft_consensus.cc:740] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:20.961031 1767 consensus_queue.cc:260] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.961369 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3d80885fcc642b8a75da30cd4abecbd" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:20.961915 1586 leader_election.cc:343] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.962381 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3d80885fcc642b8a75da30cd4abecbd" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:20.962610 1505 raft_consensus.cc:3060] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:20.964000 1767 ts_tablet_manager.cc:1434] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251024 19:48:20.964282 1767 tablet_bootstrap.cc:492] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:20.964834 1776 log.cc:1199] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/94fc8c5716af47faa4e831f05fce8610
I20251024 19:48:20.965214 1776 ts_tablet_manager.cc:1950] T 94fc8c5716af47faa4e831f05fce8610 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:20.965811 1505 raft_consensus.cc:2468] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:20.966275 1586 leader_election.cc:304] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:20.966815 2857 raft_consensus.cc:493] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:20.966929 2857 raft_consensus.cc:515] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:20.967453 2857 leader_election.cc:290] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:20.967758 3137 raft_consensus.cc:2804] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:20.967866 3137 raft_consensus.cc:697] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:20.967959 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65424b00ab9c454f9584048caba83e6a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:20.968107 3137 consensus_queue.cc:237] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:20.968454 1444 leader_election.cc:343] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.969154 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65424b00ab9c454f9584048caba83e6a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:20.969758 1447 leader_election.cc:343] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:20.969871 1447 leader_election.cc:304] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:20.970289 2857 raft_consensus.cc:2749] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:20.977845 1775 ts_tablet_manager.cc:1929] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:20.981833 1774 raft_consensus.cc:2243] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.981993 1775 log.cc:1199] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/62667092d7e244cca1020ec220ea9124
I20251024 19:48:20.982247 1774 raft_consensus.cc:2272] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.982287 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 937) took 3402 ms. Trace:
I20251024 19:48:20.982506 1775 ts_tablet_manager.cc:1950] T 62667092d7e244cca1020ec220ea9124 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:20.982496 1776 rpcz_store.cc:276] 1024 19:48:17.579362 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.579428 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:20.783594 (+3204166us) ts_tablet_manager.cc:1151] Deleting tablet 94fc8c5716af47faa4e831f05fce8610
1024 19:48:20.783602 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.901224 (+117622us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.905269 (+ 4045us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.982260 (+ 76991us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3203618}
I20251024 19:48:20.983090 1774 ts_tablet_manager.cc:1916] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.983043 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 94fc8c5716af47faa4e831f05fce8610 (table default.Table_023e47eac6c24e328d5a01fcf3b2f579 [id=048d560cd32a44c5abf12bde27fe35c7]) successfully deleted
I20251024 19:48:20.983496 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 941) took 3312 ms. Trace:
I20251024 19:48:20.983649 1776 rpcz_store.cc:276] 1024 19:48:17.670977 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.671031 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:20.983352 (+3312321us) ts_tablet_manager.cc:1151] Deleting tablet b6803d23a4714934b85190886611d8d1
1024 19:48:20.983365 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.983479 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3312148}
I20251024 19:48:20.983784 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 914) took 3830 ms. Trace:
W20251024 19:48:20.984329 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b6803d23a4714934b85190886611d8d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6803d23a4714934b85190886611d8d1 already in progress: creating tablet
I20251024 19:48:20.984738 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 942) took 3295 ms. Trace:
I20251024 19:48:20.984752 1775 rpcz_store.cc:276] 1024 19:48:17.153136 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.153186 (+ 50us) service_pool.cc:225] Handling call
1024 19:48:20.951628 (+3798442us) ts_tablet_manager.cc:1151] Deleting tablet 62667092d7e244cca1020ec220ea9124
1024 19:48:20.951637 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.955611 (+ 3974us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.977726 (+ 22115us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.983765 (+ 6039us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3798297}
I20251024 19:48:20.984901 1776 rpcz_store.cc:276] 1024 19:48:17.689084 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.689146 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:20.984615 (+3295469us) ts_tablet_manager.cc:1151] Deleting tablet 2c96c11697514af6bf4e80a67aadf748
1024 19:48:20.984626 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.984718 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3295278}
W20251024 19:48:20.985416 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2c96c11697514af6bf4e80a67aadf748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c96c11697514af6bf4e80a67aadf748 already in progress: creating tablet
I20251024 19:48:20.985514 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 62667092d7e244cca1020ec220ea9124 (table default.Table_06b07944861041f8a66001a6555d7659 [id=96c4981300dc48e6bf55fc633164ed63]) successfully deleted
I20251024 19:48:20.985918 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 947) took 3261 ms. Trace:
I20251024 19:48:20.985982 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 915) took 3810 ms. Trace:
I20251024 19:48:20.986066 1776 rpcz_store.cc:276] 1024 19:48:17.724325 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.724382 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:20.985786 (+3261404us) ts_tablet_manager.cc:1151] Deleting tablet b724eadbb86e4eeab4ba8dfd565290ff
1024 19:48:20.985798 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.985901 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3261197}
I20251024 19:48:20.986459 1775 rpcz_store.cc:276] 1024 19:48:17.175445 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.175515 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:20.985859 (+3810344us) ts_tablet_manager.cc:1151] Deleting tablet b8d30b2fdeec410fa582a05db0968ebb
1024 19:48:20.985870 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.985964 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3810164}
W20251024 19:48:20.987094 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b8d30b2fdeec410fa582a05db0968ebb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8d30b2fdeec410fa582a05db0968ebb already in progress: creating tablet
I20251024 19:48:20.987514 1775 tablet_replica.cc:333] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:20.990875 1188 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet b632acc1348e49f8a1eea019c007780c (table default.Table_361840e201d943c18014ec71a255cc9c [id=dc00ee30038e49fb9f86499cfb985a0e]). Expected version 2 got 0
I20251024 19:48:20.991627 1188 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 2d01572e51694d9bb58c3a59d4205102 (table default.Table_bf15d3c700774490b1860d3d8865a95e [id=2ea13e78137d47429d7cbe374d2113ef]). Expected version 1 got 0
I20251024 19:48:20.993086 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 23d4dd0b42d343cb9354ac698a8c761f with delete_type TABLET_DATA_DELETED (replaced by tablet fce7de9523fb4878a58aa8aef0c6b182) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:20.993501 1775 raft_consensus.cc:2243] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:20.994021 1775 raft_consensus.cc:2272] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:20.994875 1775 ts_tablet_manager.cc:1916] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:20.991166 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 4ee9fb177b4a4dbc8fcd2c9f0b6ee699 (DEFAULT_TABLE table=default.Table_4c91e1a2322c48b69d7b31b4dbdbac64 [id=a1d8deb2bce24950a5c6361d062ac1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.005533 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee9fb177b4a4dbc8fcd2c9f0b6ee699. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.007632 1766 tablet_bootstrap.cc:492] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:21.007881 1766 ts_tablet_manager.cc:1403] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.006s
I20251024 19:48:21.008569 1775 ts_tablet_manager.cc:1929] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.008752 1766 raft_consensus.cc:359] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.008998 1766 raft_consensus.cc:385] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.009109 1766 raft_consensus.cc:740] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.009045 1775 log.cc:1199] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/9489a01434a24d9ca94791252950725c
I20251024 19:48:21.009675 1775 ts_tablet_manager.cc:1950] T 9489a01434a24d9ca94791252950725c P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:20.997773 1774 ts_tablet_manager.cc:1929] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.009929 1774 log.cc:1199] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/3324755a5a184a609fae5f7a25d58e4c
I20251024 19:48:21.010313 1774 ts_tablet_manager.cc:1950] T 3324755a5a184a609fae5f7a25d58e4c P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:21.010596 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 906) took 3862 ms. Trace:
I20251024 19:48:21.010725 1775 rpcz_store.cc:276] 1024 19:48:17.148080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.205114 (+ 57034us) service_pool.cc:225] Handling call
1024 19:48:20.987444 (+3782330us) ts_tablet_manager.cc:1151] Deleting tablet 9489a01434a24d9ca94791252950725c
1024 19:48:20.987469 (+ 25us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.000259 (+ 12790us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.008470 (+ 8211us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.010574 (+ 2104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3782160}
I20251024 19:48:21.011104 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 906) took 1754 ms. Trace:
I20251024 19:48:21.011195 1774 rpcz_store.cc:276] 1024 19:48:19.256876 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.256938 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:20.931368 (+1674430us) ts_tablet_manager.cc:1151] Deleting tablet 3324755a5a184a609fae5f7a25d58e4c
1024 19:48:20.931376 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:20.993925 (+ 62549us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:20.997612 (+ 3687us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.011082 (+ 13470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1674218}
I20251024 19:48:21.011294 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 9489a01434a24d9ca94791252950725c (table default.Table_1c89d244042a4852a9cb4113f6726332 [id=ba207dd9990d4a0e85eaefbb4908dd02]) successfully deleted
I20251024 19:48:21.011539 1774 tablet_replica.cc:333] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:21.011593 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 3324755a5a184a609fae5f7a25d58e4c (table default.Table_2f543df6b1194a7195ecd656134e4b17 [id=e908b8ad13614c3c90e0f8aaafd0421f]) successfully deleted
I20251024 19:48:21.011731 1774 raft_consensus.cc:2243] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.011890 1774 raft_consensus.cc:2272] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.012050 1775 tablet_replica.cc:333] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.012266 1775 raft_consensus.cc:2243] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.012447 1775 raft_consensus.cc:2272] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.013000 1775 ts_tablet_manager.cc:1916] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.013193 1766 consensus_queue.cc:260] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.014919 1774 ts_tablet_manager.cc:1916] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.016826 1766 ts_tablet_manager.cc:1434] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251024 19:48:21.018232 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 4ee9fb177b4a4dbc8fcd2c9f0b6ee699 (DEFAULT_TABLE table=default.Table_4c91e1a2322c48b69d7b31b4dbdbac64 [id=a1d8deb2bce24950a5c6361d062ac1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.018648 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee9fb177b4a4dbc8fcd2c9f0b6ee699. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.022054 1774 ts_tablet_manager.cc:1929] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.022178 1774 log.cc:1199] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/6f89a3bbeb9b4d57a49078729110d951
I20251024 19:48:21.022554 1774 ts_tablet_manager.cc:1950] T 6f89a3bbeb9b4d57a49078729110d951 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:21.022747 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 27ff5499d30a4f178d8aceff7d0d8106 (DEFAULT_TABLE table=default.Table_b50a1dbc517648769e34a3655fcef35f [id=8392c502d817401ea08314a1346ed8bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.023182 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ff5499d30a4f178d8aceff7d0d8106. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:20.987107 1768 tablet_bootstrap.cc:492] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:21.023720 1768 ts_tablet_manager.cc:1403] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.000s
I20251024 19:48:21.024544 1768 raft_consensus.cc:359] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.024987 1768 raft_consensus.cc:385] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.025058 1768 raft_consensus.cc:740] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.025269 1768 consensus_queue.cc:260] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.025838 1768 ts_tablet_manager.cc:1434] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:21.029034 1766 tablet_bootstrap.cc:492] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:21.030614 1766 tablet_bootstrap.cc:654] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.032544 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 27ff5499d30a4f178d8aceff7d0d8106 (DEFAULT_TABLE table=default.Table_b50a1dbc517648769e34a3655fcef35f [id=8392c502d817401ea08314a1346ed8bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.036931 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ff5499d30a4f178d8aceff7d0d8106. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.037382 1341 tablet_service.cc:1505] Processing CreateTablet for tablet c084db163d874639b4385f41e9efaa18 (DEFAULT_TABLE table=default.Table_4834ef486a9b47a9976759474b0b2ca1 [id=33bdf13199244860a3765a36e3f38b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.037827 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c084db163d874639b4385f41e9efaa18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.037559 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ad2a772fc15e439bba0dfae4c45da5cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:21.038635 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad2a772fc15e439bba0dfae4c45da5cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.041354 1768 tablet_bootstrap.cc:492] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:21.043798 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d26e02ad2945b39cf27738ed5b7cf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.045092 1768 tablet_bootstrap.cc:654] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.045392 1188 catalog_manager.cc:5649] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:21.046281 1188 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 2d01572e51694d9bb58c3a59d4205102 (table default.Table_bf15d3c700774490b1860d3d8865a95e [id=2ea13e78137d47429d7cbe374d2113ef]). Expected version 1 got 0
I20251024 19:48:21.047066 1188 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet d6de2c21796f4bc4904f88b654c3baec (table default.Table_1a079b981b694617b3baf9897dfb37ae [id=18ec057e920e4c21ae8cbe298058b6f3]). Expected version 2 got 0
I20251024 19:48:21.050256 1775 ts_tablet_manager.cc:1929] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.050496 1775 log.cc:1199] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/639836c656f44a819036f4d00e055bd9
I20251024 19:48:21.050891 1775 ts_tablet_manager.cc:1950] T 639836c656f44a819036f4d00e055bd9 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:21.051656 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 905) took 3903 ms. Trace:
I20251024 19:48:21.052490 1766 tablet_bootstrap.cc:492] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:21.052721 1766 ts_tablet_manager.cc:1403] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20251024 19:48:21.053345 1766 raft_consensus.cc:359] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:21.053601 1766 raft_consensus.cc:385] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.053869 1766 raft_consensus.cc:740] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.054235 1766 consensus_queue.cc:260] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:21.054843 1766 ts_tablet_manager.cc:1434] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:21.055142 1766 tablet_bootstrap.cc:492] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:21.056457 1766 tablet_bootstrap.cc:654] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.057178 1775 rpcz_store.cc:276] 1024 19:48:17.147817 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.206280 (+ 58463us) service_pool.cc:225] Handling call
1024 19:48:21.012005 (+3805725us) ts_tablet_manager.cc:1151] Deleting tablet 639836c656f44a819036f4d00e055bd9
1024 19:48:21.012015 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.033867 (+ 21852us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.050131 (+ 16264us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.051636 (+ 1505us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3805559}
I20251024 19:48:21.058204 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 916) took 3687 ms. Trace:
I20251024 19:48:21.058871 1775 rpcz_store.cc:276] 1024 19:48:17.370621 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.370679 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:21.058047 (+3687368us) ts_tablet_manager.cc:1151] Deleting tablet eb6b7df6176e4bccacf7a9059cee5477
1024 19:48:21.058059 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.058181 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3687111}
I20251024 19:48:21.059257 1775 tablet_replica.cc:333] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.060186 3208 raft_consensus.cc:493] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.060333 3208 raft_consensus.cc:515] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.060968 3208 leader_election.cc:290] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:21.061811 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6424b2347ef642feb38a9596f2d2be1d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:21.062562 1768 tablet_bootstrap.cc:492] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:21.062809 1768 ts_tablet_manager.cc:1403] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20251024 19:48:21.063649 1768 raft_consensus.cc:359] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.063971 1768 raft_consensus.cc:385] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.063984 1775 raft_consensus.cc:2243] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.064070 1768 raft_consensus.cc:740] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.064225 1775 raft_consensus.cc:2272] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.064337 1768 consensus_queue.cc:260] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.064553 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6424b2347ef642feb38a9596f2d2be1d" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:21.064797 1505 raft_consensus.cc:2393] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:21.064822 1775 ts_tablet_manager.cc:1916] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.065068 1768 ts_tablet_manager.cc:1434] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:21.066151 1308 leader_election.cc:304] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.066551 3208 raft_consensus.cc:3060] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:21.067312 1768 tablet_bootstrap.cc:492] T 1c92b080805845c6af8f1cb160110915 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:21.068835 1768 tablet_bootstrap.cc:654] T 1c92b080805845c6af8f1cb160110915 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.069187 3208 raft_consensus.cc:2749] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.016583 1767 tablet_bootstrap.cc:654] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.018723 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 4ee9fb177b4a4dbc8fcd2c9f0b6ee699 (DEFAULT_TABLE table=default.Table_4c91e1a2322c48b69d7b31b4dbdbac64 [id=a1d8deb2bce24950a5c6361d062ac1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.070688 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee9fb177b4a4dbc8fcd2c9f0b6ee699. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.032294 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 27ff5499d30a4f178d8aceff7d0d8106 (DEFAULT_TABLE table=default.Table_b50a1dbc517648769e34a3655fcef35f [id=8392c502d817401ea08314a1346ed8bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.074386 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ff5499d30a4f178d8aceff7d0d8106. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.079890 1766 tablet_bootstrap.cc:492] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:21.081146 1766 ts_tablet_manager.cc:1403] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.003s
I20251024 19:48:21.081462 2923 consensus_queue.cc:1048] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:21.082098 1766 raft_consensus.cc:359] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.082510 1766 raft_consensus.cc:385] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.082610 1766 raft_consensus.cc:740] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.082901 1766 consensus_queue.cc:260] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.083420 1766 ts_tablet_manager.cc:1434] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:21.083719 1766 tablet_bootstrap.cc:492] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:21.059191 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 639836c656f44a819036f4d00e055bd9 (table default.Table_6eb10ae51f3a43c2ab15f362708fba94 [id=f3080df28ee94876bb6b790cea45a4e8]) successfully deleted
W20251024 19:48:21.061863 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b724eadbb86e4eeab4ba8dfd565290ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b724eadbb86e4eeab4ba8dfd565290ff already in progress: creating tablet
I20251024 19:48:21.062556 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 907) took 1753 ms. Trace:
W20251024 19:48:21.088124 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet eb6b7df6176e4bccacf7a9059cee5477 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb6b7df6176e4bccacf7a9059cee5477 already in progress: creating tablet
I20251024 19:48:21.088236 1766 tablet_bootstrap.cc:654] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.088268 1774 rpcz_store.cc:276] 1024 19:48:19.308840 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.308898 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:21.011487 (+1702589us) ts_tablet_manager.cc:1151] Deleting tablet 6f89a3bbeb9b4d57a49078729110d951
1024 19:48:21.011499 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.018644 (+ 7145us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.021958 (+ 3314us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.062536 (+ 40578us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1702370}
I20251024 19:48:21.088776 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 908) took 1778 ms. Trace:
I20251024 19:48:21.088858 1774 rpcz_store.cc:276] 1024 19:48:19.309886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.309943 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:21.088624 (+1778681us) ts_tablet_manager.cc:1151] Deleting tablet 346cb0bb8a0d43f7a15b788a303a6bcf
1024 19:48:21.088638 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.088757 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1778485}
I20251024 19:48:21.089223 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 909) took 1777 ms. Trace:
I20251024 19:48:21.089295 1774 rpcz_store.cc:276] 1024 19:48:19.311561 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.311617 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:21.089097 (+1777480us) ts_tablet_manager.cc:1151] Deleting tablet af918843c74940888c709d9a05aa6e0f
1024 19:48:21.089108 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.089206 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1777288}
I20251024 19:48:21.089329 1613 tablet_service.cc:1505] Processing CreateTablet for tablet c084db163d874639b4385f41e9efaa18 (DEFAULT_TABLE table=default.Table_4834ef486a9b47a9976759474b0b2ca1 [id=33bdf13199244860a3765a36e3f38b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.089594 1774 tablet_replica.cc:333] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:21.089768 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c084db163d874639b4385f41e9efaa18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.089808 1774 raft_consensus.cc:2243] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.090025 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 2cfce0cbed5646429b2d07d7adcadcdb (DEFAULT_TABLE table=default.Table_692697818af34700ad79bbc08b8deb43 [id=e16de4a847ca408ebf0dd1318c375eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.090209 1774 raft_consensus.cc:2272] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.090371 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cfce0cbed5646429b2d07d7adcadcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.090811 1774 ts_tablet_manager.cc:1916] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.091567 1775 ts_tablet_manager.cc:1929] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.091660 1775 log.cc:1199] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/3324755a5a184a609fae5f7a25d58e4c
I20251024 19:48:21.091977 1775 ts_tablet_manager.cc:1950] T 3324755a5a184a609fae5f7a25d58e4c P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:21.092717 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 917) took 3704 ms. Trace:
I20251024 19:48:21.092804 1775 rpcz_store.cc:276] 1024 19:48:17.388308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.388368 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:21.059204 (+3670836us) ts_tablet_manager.cc:1151] Deleting tablet 3324755a5a184a609fae5f7a25d58e4c
1024 19:48:21.059214 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.076268 (+ 17054us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.091468 (+ 15200us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.092695 (+ 1227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3670584}
I20251024 19:48:21.093094 1775 tablet_replica.cc:333] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.093278 1775 raft_consensus.cc:2243] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:21.093892 1616 tablet_service.cc:1505] Processing CreateTablet for tablet f0ac8d6e3649423e95b0c8be87adc7aa (DEFAULT_TABLE table=default.Table_c89ac6d4e142431a9d712dc0f1e32987 [id=9e23efebcb1649c2808264507e2744d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.094240 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0ac8d6e3649423e95b0c8be87adc7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.089784 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 3753358603b6458ebdb5d59f16363629 (DEFAULT_TABLE table=default.Table_fe16d951e74c460f901b1898a2a06551 [id=d567295f6cd744a28102a153c9f689c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.095938 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3753358603b6458ebdb5d59f16363629. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.099267 1775 raft_consensus.cc:2272] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.099627 1344 tablet_service.cc:1505] Processing CreateTablet for tablet f0ac8d6e3649423e95b0c8be87adc7aa (DEFAULT_TABLE table=default.Table_c89ac6d4e142431a9d712dc0f1e32987 [id=9e23efebcb1649c2808264507e2744d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.099888 1775 ts_tablet_manager.cc:1916] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.100098 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0ac8d6e3649423e95b0c8be87adc7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.101935 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 3753358603b6458ebdb5d59f16363629 (DEFAULT_TABLE table=default.Table_fe16d951e74c460f901b1898a2a06551 [id=d567295f6cd744a28102a153c9f689c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.102317 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3753358603b6458ebdb5d59f16363629. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.108536 1766 tablet_bootstrap.cc:492] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:21.108815 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 04ae62c9bf6f44108d57d0760283d063
I20251024 19:48:21.108808 1766 ts_tablet_manager.cc:1403] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.004s
I20251024 19:48:21.109618 1766 raft_consensus.cc:359] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.109786 1766 raft_consensus.cc:385] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.109813 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7ecf3276437a43cf84a47a0e66553e0a
I20251024 19:48:21.109839 1766 raft_consensus.cc:740] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.110031 1766 consensus_queue.cc:260] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.110510 1766 ts_tablet_manager.cc:1434] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:21.110798 1766 tablet_bootstrap.cc:492] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:21.112824 1766 tablet_bootstrap.cc:654] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.116225 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet 04ae62c9bf6f44108d57d0760283d063 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.116225 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ecf3276437a43cf84a47a0e66553e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.117038 3141 raft_consensus.cc:493] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.117182 3141 raft_consensus.cc:515] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.117870 3141 leader_election.cc:290] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:21.118391 1775 ts_tablet_manager.cc:1929] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.118484 1775 log.cc:1199] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/b240965605904533bd2af99f61aaef26
I20251024 19:48:21.118835 1775 ts_tablet_manager.cc:1950] T b240965605904533bd2af99f61aaef26 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:21.119699 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 918) took 3722 ms. Trace:
I20251024 19:48:21.119773 1775 rpcz_store.cc:276] 1024 19:48:17.396973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.397034 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:21.093053 (+3696019us) ts_tablet_manager.cc:1151] Deleting tablet b240965605904533bd2af99f61aaef26
1024 19:48:21.093061 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.108141 (+ 15080us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.118289 (+ 10148us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.119665 (+ 1376us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3695810}
I20251024 19:48:21.120049 1775 tablet_replica.cc:333] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.120214 1775 raft_consensus.cc:2243] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.120338 1775 raft_consensus.cc:2272] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.120824 1775 ts_tablet_manager.cc:1916] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.122473 1768 tablet_bootstrap.cc:492] T 1c92b080805845c6af8f1cb160110915 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:21.122618 1768 ts_tablet_manager.cc:1403] T 1c92b080805845c6af8f1cb160110915 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.000s
I20251024 19:48:21.123526 1768 raft_consensus.cc:359] T 1c92b080805845c6af8f1cb160110915 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.123715 1768 raft_consensus.cc:385] T 1c92b080805845c6af8f1cb160110915 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.123793 1768 raft_consensus.cc:740] T 1c92b080805845c6af8f1cb160110915 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.124014 1768 consensus_queue.cc:260] T 1c92b080805845c6af8f1cb160110915 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.124702 1768 ts_tablet_manager.cc:1434] T 1c92b080805845c6af8f1cb160110915 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:21.125223 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28005301f9de48e5826f46c79aeb5fe5" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:21.125413 1511 raft_consensus.cc:2468] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:21.126739 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28005301f9de48e5826f46c79aeb5fe5" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:21.127615 1586 leader_election.cc:304] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:21.128110 3163 raft_consensus.cc:2804] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:21.128196 3163 raft_consensus.cc:493] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:21.128266 3163 raft_consensus.cc:3060] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:21.129813 1768 tablet_bootstrap.cc:492] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:21.129863 1473 tablet_service.cc:1505] Processing CreateTablet for tablet c084db163d874639b4385f41e9efaa18 (DEFAULT_TABLE table=default.Table_4834ef486a9b47a9976759474b0b2ca1 [id=33bdf13199244860a3765a36e3f38b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.129983 1487 tablet_service.cc:1505] Processing CreateTablet for tablet f0ac8d6e3649423e95b0c8be87adc7aa (DEFAULT_TABLE table=default.Table_c89ac6d4e142431a9d712dc0f1e32987 [id=9e23efebcb1649c2808264507e2744d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.130414 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c084db163d874639b4385f41e9efaa18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.129863 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 2cfce0cbed5646429b2d07d7adcadcdb (DEFAULT_TABLE table=default.Table_692697818af34700ad79bbc08b8deb43 [id=e16de4a847ca408ebf0dd1318c375eb7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:21.134008 1586 leader_election.cc:343] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.134205 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cfce0cbed5646429b2d07d7adcadcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.136456 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 04ae62c9bf6f44108d57d0760283d063 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.136690 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ecf3276437a43cf84a47a0e66553e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.137240 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ecf3276437a43cf84a47a0e66553e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.137676 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 45cff9bcc4924e0081518c284f074940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.138170 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0ac8d6e3649423e95b0c8be87adc7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.148061 3163 raft_consensus.cc:515] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.148764 3163 leader_election.cc:290] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:21.149024 1774 ts_tablet_manager.cc:1929] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:21.131464 1768 tablet_bootstrap.cc:654] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.155547 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28005301f9de48e5826f46c79aeb5fe5" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:21.155728 1504 raft_consensus.cc:3060] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:21.158365 1504 raft_consensus.cc:2468] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:21.159049 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28005301f9de48e5826f46c79aeb5fe5" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:21.159776 1586 leader_election.cc:343] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.160022 1586 leader_election.cc:304] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:21.160784 3163 raft_consensus.cc:2804] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:21.160898 3163 raft_consensus.cc:697] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:21.161036 1774 log.cc:1199] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/8a4cb86bc5e745d5b4ee932f0a38b3c7
I20251024 19:48:21.161141 3163 consensus_queue.cc:237] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.161689 1774 ts_tablet_manager.cc:1950] T 8a4cb86bc5e745d5b4ee932f0a38b3c7 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:21.162186 1775 ts_tablet_manager.cc:1929] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.163410 1775 log.cc:1199] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/6f630fd412ca4c4f8570359c1678e6af
I20251024 19:48:21.163938 1775 ts_tablet_manager.cc:1950] T 6f630fd412ca4c4f8570359c1678e6af P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:21.164461 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 932) took 1567 ms. Trace:
I20251024 19:48:21.164860 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 919) took 3767 ms. Trace:
I20251024 19:48:21.167297 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 30bce965315d43948ecb8282c829c557
I20251024 19:48:21.167807 1183 catalog_manager.cc:6719] ReplaceTablet: tablet 30bce965315d43948ecb8282c829c557 deleted and replaced by tablet 043fac06e6324032a61c42a10d51ea22
I20251024 19:48:21.168278 1775 rpcz_store.cc:276] 1024 19:48:17.397780 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.397841 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:21.120005 (+3722164us) ts_tablet_manager.cc:1151] Deleting tablet 6f630fd412ca4c4f8570359c1678e6af
1024 19:48:21.120014 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.129116 (+ 9102us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.162053 (+ 32937us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.164838 (+ 2785us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3721995}
I20251024 19:48:21.168812 1775 tablet_replica.cc:333] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.169036 1775 raft_consensus.cc:2243] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.169215 1775 raft_consensus.cc:2272] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.170176 1775 ts_tablet_manager.cc:1916] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.170934 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8d0cf72d8a64479cb61a8f1317f392c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:21.171551 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d0cf72d8a64479cb61a8f1317f392c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.173700 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 672b571081ea4cd78028b6a25625103c
I20251024 19:48:21.176002 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_243a93945dd5416e94a881c0e85e5875" } new_table_name: "default.Table_ac91d48ce11a4e0cbed30752c680d6eb" modify_external_catalogs: true
I20251024 19:48:21.177335 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet e0b57403cdf947b390a44a9889f2a2c3
I20251024 19:48:21.178028 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253
I20251024 19:48:21.178259 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3a99e9b3fe044c7eb14fa9bffa903873
I20251024 19:48:21.179046 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet e0b57403cdf947b390a44a9889f2a2c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.178974 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f5c4c9448ed5406cbc1c615cdcca8781"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:21.179395 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet 672b571081ea4cd78028b6a25625103c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:21.179522 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5c4c9448ed5406cbc1c615cdcca8781 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.179638 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 30bce965315d43948ecb8282c829c557 with delete_type TABLET_DATA_DELETED (replaced by tablet 043fac06e6324032a61c42a10d51ea22) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.185755 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 59eca6bb1dd84c67a7e53131abaa937a
I20251024 19:48:21.061811 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 948) took 3331 ms. Trace:
I20251024 19:48:21.187204 1776 rpcz_store.cc:276] 1024 19:48:17.730688 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.730738 (+ 50us) service_pool.cc:225] Handling call
1024 19:48:21.061647 (+3330909us) ts_tablet_manager.cc:1151] Deleting tablet f4e812c43ade4831af50825474976c8e
1024 19:48:21.061661 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.061789 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3330698}
I20251024 19:48:21.194015 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 949) took 3343 ms. Trace:
I20251024 19:48:21.194121 1776 rpcz_store.cc:276] 1024 19:48:17.850495 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.850553 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:21.193829 (+3343276us) ts_tablet_manager.cc:1151] Deleting tablet aeeaf256b1404653b690c712d9ef7a6d
1024 19:48:21.193842 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.193990 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3342971}
I20251024 19:48:21.195217 1776 tablet_replica.cc:333] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:21.195433 1776 raft_consensus.cc:2243] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:21.195777 1776 raft_consensus.cc:2272] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.200510 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a99e9b3fe044c7eb14fa9bffa903873 with delete_type TABLET_DATA_DELETED (replaced by tablet b4b7defa4cac4f62988eb0a0b1227997) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.200603 1186 catalog_manager.cc:6719] ReplaceTablet: tablet 3a99e9b3fe044c7eb14fa9bffa903873 deleted and replaced by tablet b4b7defa4cac4f62988eb0a0b1227997
I20251024 19:48:21.200824 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet 1edc55caab904722bf19a1c71df601cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.200980 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0ba3b1e96f494fb0821fc035f4b67162"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:21.201094 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet d840888183ec4ef7bb245363c5bac322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.201246 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 with delete_type TABLET_DATA_DELETED (replaced by tablet 42c6fbc3067d433489230906816fc202) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.201444 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet 59eca6bb1dd84c67a7e53131abaa937a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:21.201999 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ba3b1e96f494fb0821fc035f4b67162 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.204015 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_74e4f24f12db403686548924c83c5191" } new_table_name: "default.Table_06ba8e65390c454fb0c4168ec0872485" modify_external_catalogs: true
W20251024 19:48:21.204277 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9367428428648f7ae07722ea09d70a7 consensus state has no leader
I20251024 19:48:21.207453 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet b74415cf7c7e47a696b454a5d44d8017 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:21.208194 1775 ts_tablet_manager.cc:1929] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.208179 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f7dee67353c1481485de2f883d0bf7bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:21.208318 1775 log.cc:1199] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/cb0405765c6146cbbb187af96a60be2e
I20251024 19:48:21.208683 1775 ts_tablet_manager.cc:1950] T cb0405765c6146cbbb187af96a60be2e P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
W20251024 19:48:21.208741 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7dee67353c1481485de2f883d0bf7bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.209407 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 920) took 3811 ms. Trace:
I20251024 19:48:21.209483 1775 rpcz_store.cc:276] 1024 19:48:17.398432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.398478 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:21.168768 (+3770290us) ts_tablet_manager.cc:1151] Deleting tablet cb0405765c6146cbbb187af96a60be2e
1024 19:48:21.168777 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.199292 (+ 30515us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.208063 (+ 8771us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.209387 (+ 1324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3770137}
I20251024 19:48:21.211099 1191 catalog_manager.cc:6719] ReplaceTablet: tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 deleted and replaced by tablet 42c6fbc3067d433489230906816fc202
I20251024 19:48:21.211169 2857 raft_consensus.cc:493] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.211330 2857 raft_consensus.cc:515] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.212105 2857 leader_election.cc:290] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:21.212972 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "250f106664be4b5aaa330dbb4e43cb7b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:21.213644 1444 leader_election.cc:343] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:21.214406 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f4e812c43ade4831af50825474976c8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4e812c43ade4831af50825474976c8e already in progress: creating tablet
I20251024 19:48:21.215713 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8f9e49a43cb94787a02a10d3017b4e75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:21.216392 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f9e49a43cb94787a02a10d3017b4e75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.218056 1776 ts_tablet_manager.cc:1916] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.218425 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2f3f9f0797ae454eb0fd2975c56cadfd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:21.219338 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f3f9f0797ae454eb0fd2975c56cadfd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:21.222266 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet aeeaf256b1404653b690c712d9ef7a6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aeeaf256b1404653b690c712d9ef7a6d already in progress: creating tablet
I20251024 19:48:21.223227 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet 59956dad06064024ab171631ef6c1dca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.223470 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet f3d80885fcc642b8a75da30cd4abecbd with delete_type TABLET_DATA_DELETED (replaced by tablet 780442b310e6409498e6bbe4256280a4) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.220474 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_72458dd9d8744a1ba775cfdcd2cc6e8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:21.224373 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 3324755a5a184a609fae5f7a25d58e4c (table default.Table_2f543df6b1194a7195ecd656134e4b17 [id=e908b8ad13614c3c90e0f8aaafd0421f]) successfully deleted
W20251024 19:48:21.224704 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 48682.9us
I20251024 19:48:21.225646 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 9026d9e801294dd2bfdb9c90e45a0990 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.225793 1775 tablet_replica.cc:333] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.226054 1775 raft_consensus.cc:2243] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.226264 1775 raft_consensus.cc:2272] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.226115 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet 28005301f9de48e5826f46c79aeb5fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.227425 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet b240965605904533bd2af99f61aaef26 (table default.Table_a1366192b3134e3abfc231729cb42344 [id=ae6b320a09934ac5873d3b35c33f2c65]) successfully deleted
I20251024 19:48:21.228137 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 6f630fd412ca4c4f8570359c1678e6af (table default.Table_a5d11f93359546078ef5becf10539131 [id=e30a8747e9a6453e9567626b55ccfc1a]) successfully deleted
I20251024 19:48:21.229562 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet cb0405765c6146cbbb187af96a60be2e (table default.Table_3f7f7b5dfa674b19ad19c339c619326c [id=7faf1f110af34aa8939ad42e64f60011]) successfully deleted
I20251024 19:48:21.229873 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 6f89a3bbeb9b4d57a49078729110d951 (table default.Table_f5d35eb71b3948ffbc2a9c015823e67f [id=c469aebfcbde437abe1636f85c39d6fb]) successfully deleted
I20251024 19:48:21.231027 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6b8a2c3a14404cab96880c1a8630f56c" } new_table_name: "default.Table_e26d99c098b743d2b81cfe7db40784af" modify_external_catalogs: true
I20251024 19:48:21.232828 1775 ts_tablet_manager.cc:1916] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:21.241374 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72458dd9d8744a1ba775cfdcd2cc6e8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.245540 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0a3260e932f447998a886cb5e967fd46" } new_table_name: "default.Table_d3ffdf5cbfd84551b1e314d4a772be1a" modify_external_catalogs: true
I20251024 19:48:21.253008 2857 raft_consensus.cc:493] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.253176 2857 raft_consensus.cc:515] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.254175 2857 leader_election.cc:290] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:21.254519 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da7d38b63708427982591da1014aca81" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:21.255401 1444 leader_election.cc:343] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.259379 1767 tablet_bootstrap.cc:492] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:21.259536 1767 ts_tablet_manager.cc:1403] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.295s user 0.002s sys 0.002s
I20251024 19:48:21.168219 1768 tablet_bootstrap.cc:492] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:21.182559 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 2cfce0cbed5646429b2d07d7adcadcdb (DEFAULT_TABLE table=default.Table_692697818af34700ad79bbc08b8deb43 [id=e16de4a847ca408ebf0dd1318c375eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.214282 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "250f106664be4b5aaa330dbb4e43cb7b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:21.260239 1768 ts_tablet_manager.cc:1403] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.131s user 0.002s sys 0.001s
I20251024 19:48:21.255954 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da7d38b63708427982591da1014aca81" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:21.260978 1447 leader_election.cc:343] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.261094 1447 leader_election.cc:304] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.261054 1768 raft_consensus.cc:359] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.261226 1768 raft_consensus.cc:385] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.261297 1768 raft_consensus.cc:740] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
W20251024 19:48:21.261528 1447 leader_election.cc:343] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.261485 1768 consensus_queue.cc:260] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.261636 1447 leader_election.cc:304] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.262028 2857 raft_consensus.cc:2749] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.262030 2858 raft_consensus.cc:2749] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.262787 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cfce0cbed5646429b2d07d7adcadcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.266681 1768 ts_tablet_manager.cc:1434] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251024 19:48:21.271270 1766 tablet_bootstrap.cc:492] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:21.271423 1766 ts_tablet_manager.cc:1403] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.161s user 0.033s sys 0.030s
I20251024 19:48:21.272326 1766 raft_consensus.cc:359] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.272478 1768 tablet_bootstrap.cc:492] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:21.272564 1766 raft_consensus.cc:385] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.272621 1766 raft_consensus.cc:740] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.273005 1766 consensus_queue.cc:260] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.273718 1775 ts_tablet_manager.cc:1929] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.273851 1775 log.cc:1199] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/844638a6b6944d0394df3e78a8ea51d6
I20251024 19:48:21.274271 1775 ts_tablet_manager.cc:1950] T 844638a6b6944d0394df3e78a8ea51d6 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:21.277990 3240 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ef8906492b1a4368936fac95c11e1f73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251024 19:48:21.280467 1766 ts_tablet_manager.cc:1434] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251024 19:48:21.281313 1766 tablet_bootstrap.cc:492] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:21.284758 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 921) took 3883 ms. Trace:
I20251024 19:48:21.284870 1775 rpcz_store.cc:276] 1024 19:48:17.401227 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.401300 (+ 73us) service_pool.cc:225] Handling call
1024 19:48:21.225727 (+3824427us) ts_tablet_manager.cc:1151] Deleting tablet 844638a6b6944d0394df3e78a8ea51d6
1024 19:48:21.225740 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.246572 (+ 20832us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.273168 (+ 26596us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.284735 (+ 11567us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3824275}
I20251024 19:48:21.285202 1775 tablet_replica.cc:333] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.285389 1775 raft_consensus.cc:2243] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.285609 1775 raft_consensus.cc:2272] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.286157 1775 ts_tablet_manager.cc:1916] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.290473 1768 tablet_bootstrap.cc:654] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.294637 1766 tablet_bootstrap.cc:654] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.284778 1776 ts_tablet_manager.cc:1929] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.286712 3137 raft_consensus.cc:493] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.295011 3137 raft_consensus.cc:515] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.295672 3137 leader_election.cc:290] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:21.295545 1774 rpcz_store.cc:276] 1024 19:48:19.611555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.611615 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:21.089526 (+1477911us) ts_tablet_manager.cc:1151] Deleting tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7
1024 19:48:21.089534 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.145904 (+ 56370us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.148928 (+ 3024us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.164440 (+ 15512us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1477720}
I20251024 19:48:21.296070 1776 log.cc:1199] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/6f89a3bbeb9b4d57a49078729110d951
I20251024 19:48:21.296547 1776 ts_tablet_manager.cc:1950] T 6f89a3bbeb9b4d57a49078729110d951 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:21.296756 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9026d9e801294dd2bfdb9c90e45a0990" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:21.297376 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 957) took 3250 ms. Trace:
I20251024 19:48:21.297474 1776 rpcz_store.cc:276] 1024 19:48:18.046499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.046657 (+ 158us) service_pool.cc:225] Handling call
1024 19:48:21.195166 (+3148509us) ts_tablet_manager.cc:1151] Deleting tablet 6f89a3bbeb9b4d57a49078729110d951
1024 19:48:21.195175 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.221353 (+ 26178us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.284647 (+ 63294us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.297355 (+ 12708us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3148280}
W20251024 19:48:21.298096 1586 leader_election.cc:343] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:21.298864 1586 leader_election.cc:343] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 9026d9e801294dd2bfdb9c90e45a0990
I20251024 19:48:21.299015 1586 leader_election.cc:304] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:21.299326 3137 raft_consensus.cc:2749] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.299266 1774 tablet_replica.cc:333] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:21.300329 1774 raft_consensus.cc:2243] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.301663 1774 raft_consensus.cc:2272] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:21.304900 1586 consensus_peers.cc:597] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:21.305707 2923 consensus_queue.cc:1048] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:21.306038 1776 tablet_replica.cc:333] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:21.306267 1776 raft_consensus.cc:2243] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:21.306685 1776 raft_consensus.cc:2272] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.307369 1766 tablet_bootstrap.cc:492] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:21.307518 1766 ts_tablet_manager.cc:1403] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.003s
I20251024 19:48:21.310185 1776 ts_tablet_manager.cc:1916] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.312482 1766 raft_consensus.cc:359] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.312649 1766 raft_consensus.cc:385] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.312692 1766 raft_consensus.cc:740] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.312896 1766 consensus_queue.cc:260] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.313417 1766 ts_tablet_manager.cc:1434] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20251024 19:48:21.313723 1766 tablet_bootstrap.cc:492] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:21.313738 1775 ts_tablet_manager.cc:1929] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.314383 1775 log.cc:1199] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/6f89a3bbeb9b4d57a49078729110d951
I20251024 19:48:21.314828 1775 ts_tablet_manager.cc:1950] T 6f89a3bbeb9b4d57a49078729110d951 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:21.315529 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 922) took 3913 ms. Trace:
I20251024 19:48:21.315606 1775 rpcz_store.cc:276] 1024 19:48:17.401719 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.401763 (+ 44us) service_pool.cc:225] Handling call
1024 19:48:21.285156 (+3883393us) ts_tablet_manager.cc:1151] Deleting tablet 6f89a3bbeb9b4d57a49078729110d951
1024 19:48:21.285165 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.306748 (+ 21583us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.313627 (+ 6879us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.315511 (+ 1884us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3883254}
I20251024 19:48:21.316195 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 923) took 3912 ms. Trace:
I20251024 19:48:21.316268 1775 rpcz_store.cc:276] 1024 19:48:17.403449 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.403504 (+ 55us) service_pool.cc:225] Handling call
1024 19:48:21.316082 (+3912578us) ts_tablet_manager.cc:1151] Deleting tablet 40338a7939be4240b44dc54cc77ef020
1024 19:48:21.316091 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.316181 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3912431}
I20251024 19:48:21.317028 1366 raft_consensus.cc:3060] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:21.322444 2923 consensus_queue.cc:1048] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:21.333473 1774 ts_tablet_manager.cc:1916] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:21.340502 1586 consensus_peers.cc:597] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:21.316970 1775 tablet_replica.cc:333] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.341671 1766 tablet_bootstrap.cc:654] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.348905 2923 consensus_queue.cc:1048] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:21.367404 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 346cb0bb8a0d43f7a15b788a303a6bcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 346cb0bb8a0d43f7a15b788a303a6bcf already in progress: creating tablet
W20251024 19:48:21.367646 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 137335us
W20251024 19:48:21.367930 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet af918843c74940888c709d9a05aa6e0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af918843c74940888c709d9a05aa6e0f already in progress: creating tablet
I20251024 19:48:21.368575 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 8a4cb86bc5e745d5b4ee932f0a38b3c7 (table default.Table_5006c140e0964b569953c4f03495c697 [id=338beb9c6956484daef970b28d5e4b06]) successfully deleted
I20251024 19:48:21.369235 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 30bce965315d43948ecb8282c829c557 with delete_type TABLET_DATA_DELETED (replaced by tablet 043fac06e6324032a61c42a10d51ea22) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.369714 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_368605d650e9482682f6d88537e0f51a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:21.378139 3163 consensus_queue.cc:1048] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:21.298714 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 6f89a3bbeb9b4d57a49078729110d951 (table default.Table_f5d35eb71b3948ffbc2a9c015823e67f [id=c469aebfcbde437abe1636f85c39d6fb]) successfully deleted
W20251024 19:48:21.380492 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b39e014607b4437b255c26c92c71861 consensus state has no leader
I20251024 19:48:21.385140 1775 raft_consensus.cc:2243] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.385402 1775 raft_consensus.cc:2272] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.386057 1775 ts_tablet_manager.cc:1916] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.391083 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 672b571081ea4cd78028b6a25625103c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.391371 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 04ae62c9bf6f44108d57d0760283d063 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.393801 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet e0b57403cdf947b390a44a9889f2a2c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.395973 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 with delete_type TABLET_DATA_DELETED (replaced by tablet 42c6fbc3067d433489230906816fc202) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.396200 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 59eca6bb1dd84c67a7e53131abaa937a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.396410 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a99e9b3fe044c7eb14fa9bffa903873 with delete_type TABLET_DATA_DELETED (replaced by tablet b4b7defa4cac4f62988eb0a0b1227997) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.399199 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9d66cdbfe5824468b9e6733962e5ec85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:21.404697 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 3753358603b6458ebdb5d59f16363629 (DEFAULT_TABLE table=default.Table_fe16d951e74c460f901b1898a2a06551 [id=d567295f6cd744a28102a153c9f689c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.405141 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3753358603b6458ebdb5d59f16363629. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.408711 1775 ts_tablet_manager.cc:1929] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:21.408813 1775 log.cc:1199] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/e29abfa3bc8944aea7d1acd03e3eb897
I20251024 19:48:21.413831 1775 ts_tablet_manager.cc:1950] T e29abfa3bc8944aea7d1acd03e3eb897 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:21.414721 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 924) took 3994 ms. Trace:
I20251024 19:48:21.414891 1775 rpcz_store.cc:276] 1024 19:48:17.420304 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.421662 (+ 1358us) service_pool.cc:225] Handling call
1024 19:48:21.316925 (+3895263us) ts_tablet_manager.cc:1151] Deleting tablet e29abfa3bc8944aea7d1acd03e3eb897
1024 19:48:21.316933 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.397987 (+ 81054us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.408639 (+ 10652us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.414698 (+ 6059us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3894967}
I20251024 19:48:21.415163 1766 tablet_bootstrap.cc:492] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:21.415305 1766 ts_tablet_manager.cc:1403] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.102s user 0.006s sys 0.007s
I20251024 19:48:21.415562 1775 tablet_replica.cc:333] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.415807 1775 raft_consensus.cc:2243] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.416009 1775 raft_consensus.cc:2272] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.416054 1766 raft_consensus.cc:359] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.416219 1766 raft_consensus.cc:385] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.416280 1766 raft_consensus.cc:740] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.416689 1775 ts_tablet_manager.cc:1916] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.416479 1766 consensus_queue.cc:260] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.418329 1766 ts_tablet_manager.cc:1434] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251024 19:48:21.421253 1766 tablet_bootstrap.cc:492] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:21.421902 1767 raft_consensus.cc:359] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.422101 1767 raft_consensus.cc:385] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.422163 1767 raft_consensus.cc:740] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.423530 1775 ts_tablet_manager.cc:1929] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.423687 1775 log.cc:1199] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/94fc8c5716af47faa4e831f05fce8610
I20251024 19:48:21.425747 1775 ts_tablet_manager.cc:1950] T 94fc8c5716af47faa4e831f05fce8610 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:21.427141 1766 tablet_bootstrap.cc:654] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.369989 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet 30bce965315d43948ecb8282c829c557 with delete_type TABLET_DATA_DELETED (replaced by tablet 043fac06e6324032a61c42a10d51ea22) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.387830 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 672b571081ea4cd78028b6a25625103c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.389786 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet e0b57403cdf947b390a44a9889f2a2c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.395076 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a99e9b3fe044c7eb14fa9bffa903873 with delete_type TABLET_DATA_DELETED (replaced by tablet b4b7defa4cac4f62988eb0a0b1227997) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.395367 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 with delete_type TABLET_DATA_DELETED (replaced by tablet 42c6fbc3067d433489230906816fc202) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.395537 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 59eca6bb1dd84c67a7e53131abaa937a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:20 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.405071 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 65b8d05d649244a58958a2d50265dc3a (DEFAULT_TABLE table=default.Table_7f7db12b2ef64686bb2a1b4d4cd7ed67 [id=812cc6b349d14e56bfcefc5347f1d615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.431739 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65b8d05d649244a58958a2d50265dc3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.434576 2923 consensus_queue.cc:1048] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:21.435536 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 925) took 3690 ms. Trace:
I20251024 19:48:21.435623 1775 rpcz_store.cc:276] 1024 19:48:17.745151 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.745209 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:21.415511 (+3670302us) ts_tablet_manager.cc:1151] Deleting tablet 94fc8c5716af47faa4e831f05fce8610
1024 19:48:21.415521 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.419962 (+ 4441us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.423430 (+ 3468us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.435514 (+ 12084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3670112}
I20251024 19:48:21.436122 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 928) took 3675 ms. Trace:
I20251024 19:48:21.436206 1775 rpcz_store.cc:276] 1024 19:48:17.761017 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.761080 (+ 63us) service_pool.cc:225] Handling call
1024 19:48:21.436008 (+3674928us) ts_tablet_manager.cc:1151] Deleting tablet b6803d23a4714934b85190886611d8d1
1024 19:48:21.436017 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.436107 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3674735}
I20251024 19:48:21.444469 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 929) took 3683 ms. Trace:
I20251024 19:48:21.444595 1775 rpcz_store.cc:276] 1024 19:48:17.761135 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.761326 (+ 191us) service_pool.cc:225] Handling call
1024 19:48:21.444327 (+3683001us) ts_tablet_manager.cc:1151] Deleting tablet 2c96c11697514af6bf4e80a67aadf748
1024 19:48:21.444339 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.444443 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3682860}
I20251024 19:48:21.445094 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 935) took 3623 ms. Trace:
I20251024 19:48:21.445180 1775 rpcz_store.cc:276] 1024 19:48:17.821789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.821853 (+ 64us) service_pool.cc:225] Handling call
1024 19:48:21.444992 (+3623139us) ts_tablet_manager.cc:1151] Deleting tablet b724eadbb86e4eeab4ba8dfd565290ff
1024 19:48:21.445002 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.445077 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3621748}
I20251024 19:48:21.445698 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 936) took 3619 ms. Trace:
I20251024 19:48:21.445788 1775 rpcz_store.cc:276] 1024 19:48:17.825978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:17.826032 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:21.445515 (+3619483us) ts_tablet_manager.cc:1151] Deleting tablet f4e812c43ade4831af50825474976c8e
1024 19:48:21.445523 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.445679 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3619307}
I20251024 19:48:21.446201 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 938) took 3376 ms. Trace:
I20251024 19:48:21.446282 1775 rpcz_store.cc:276] 1024 19:48:18.070063 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.070125 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:21.446102 (+3375977us) ts_tablet_manager.cc:1151] Deleting tablet aeeaf256b1404653b690c712d9ef7a6d
1024 19:48:21.446111 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.446183 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3375794}
I20251024 19:48:21.446647 1775 tablet_replica.cc:333] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.458384 1767 consensus_queue.cc:260] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.459086 1767 ts_tablet_manager.cc:1434] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.199s user 0.003s sys 0.000s
I20251024 19:48:21.463992 1768 tablet_bootstrap.cc:492] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:21.464274 1768 ts_tablet_manager.cc:1403] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.192s user 0.003s sys 0.003s
W20251024 19:48:21.466553 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_368605d650e9482682f6d88537e0f51a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:21.466946 1176 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 82026.5us
W20251024 19:48:21.468324 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d66cdbfe5824468b9e6733962e5ec85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.468909 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 844638a6b6944d0394df3e78a8ea51d6 (table default.Table_823526f1c3164a3ba70c44dd8388fb9c [id=fc5fe814531e4110ace3fb312aefa7ac]) successfully deleted
I20251024 19:48:21.469275 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 6f89a3bbeb9b4d57a49078729110d951 (table default.Table_f5d35eb71b3948ffbc2a9c015823e67f [id=c469aebfcbde437abe1636f85c39d6fb]) successfully deleted
I20251024 19:48:21.469466 1183 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet d6de2c21796f4bc4904f88b654c3baec (table default.Table_1a079b981b694617b3baf9897dfb37ae [id=18ec057e920e4c21ae8cbe298058b6f3]). Expected version 2 got 0
W20251024 19:48:21.469877 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 40338a7939be4240b44dc54cc77ef020 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40338a7939be4240b44dc54cc77ef020 already in progress: creating tablet
I20251024 19:48:21.470312 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet e29abfa3bc8944aea7d1acd03e3eb897 (table default.Table_55339e02660843c3a8536a4b7a06831e [id=ba30503223dd4979824ec9c1a50a8ae2]) successfully deleted
I20251024 19:48:21.472280 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 65b8d05d649244a58958a2d50265dc3a (DEFAULT_TABLE table=default.Table_7f7db12b2ef64686bb2a1b4d4cd7ed67 [id=812cc6b349d14e56bfcefc5347f1d615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.472734 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65b8d05d649244a58958a2d50265dc3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.473083 1776 ts_tablet_manager.cc:1929] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.473173 1776 log.cc:1199] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/844638a6b6944d0394df3e78a8ea51d6
I20251024 19:48:21.473470 1776 ts_tablet_manager.cc:1950] T 844638a6b6944d0394df3e78a8ea51d6 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:21.474164 1768 raft_consensus.cc:359] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.474326 1768 raft_consensus.cc:385] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.474370 1768 raft_consensus.cc:740] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.475697 1768 consensus_queue.cc:260] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.480024 1366 raft_consensus.cc:3060] T 1c92b080805845c6af8f1cb160110915 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:21.480854 1768 ts_tablet_manager.cc:1434] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251024 19:48:21.481132 1768 tablet_bootstrap.cc:492] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:21.484122 3163 consensus_queue.cc:1048] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:21.490723 1767 tablet_bootstrap.cc:492] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:21.493165 1775 raft_consensus.cc:2243] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.493556 1775 pending_rounds.cc:70] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e: Trying to abort 1 pending ops.
I20251024 19:48:21.493633 1775 pending_rounds.cc:77] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429390637010944 op_type: NO_OP noop_request { }
I20251024 19:48:21.493729 1775 raft_consensus.cc:2889] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:21.493821 1775 raft_consensus.cc:2272] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.498095 3315 tablet.cc:1722] T d6de2c21796f4bc4904f88b654c3baec P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:21.500277 1768 tablet_bootstrap.cc:654] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:21.509898 1586 consensus_peers.cc:597] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:21.510166 1775 ts_tablet_manager.cc:1916] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.511799 3316 tablet.cc:1722] T d6de2c21796f4bc4904f88b654c3baec P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:21.517170 1775 ts_tablet_manager.cc:1929] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.517293 1775 log.cc:1199] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/50cea450eda9485daba40bf875a1b90c
I20251024 19:48:21.517710 1775 ts_tablet_manager.cc:1950] T 50cea450eda9485daba40bf875a1b90c P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:21.520390 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 941) took 3447 ms. Trace:
I20251024 19:48:21.520509 1775 rpcz_store.cc:276] 1024 19:48:18.073169 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.073235 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:21.446602 (+3373367us) ts_tablet_manager.cc:1151] Deleting tablet 50cea450eda9485daba40bf875a1b90c
1024 19:48:21.446611 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.493466 (+ 46855us) spinlock_profiling.cc:243] Waited 46.3 ms on lock 0x6160000b6254. stack: 00007f26c12049f6 00007f26c12047b0 00007f26c01e03ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f26cd0ca100 00007f26ce417ae3 00007f26d0fd4287 00007f26d10072d4 00007f26d0fe7d70 00007f26d0fe7afd 0000000000a8867c 00007f26c124ad7f 00007f26c124e04b 00007f26c124dec8
1024 19:48:21.513927 (+ 20461us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.517070 (+ 3143us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.520365 (+ 3295us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":106454400,"tablet-delete.queue_time_us":3373164}
I20251024 19:48:21.529917 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 946) took 3349 ms. Trace:
I20251024 19:48:21.530058 1775 rpcz_store.cc:276] 1024 19:48:18.180965 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.181020 (+ 55us) service_pool.cc:225] Handling call
1024 19:48:21.529723 (+3348703us) ts_tablet_manager.cc:1151] Deleting tablet e7d6472ac7b046d89f56618fa4ed1811
1024 19:48:21.529738 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.529881 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3348505}
I20251024 19:48:21.536288 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 94fc8c5716af47faa4e831f05fce8610 (table default.Table_023e47eac6c24e328d5a01fcf3b2f579 [id=048d560cd32a44c5abf12bde27fe35c7]) successfully deleted
I20251024 19:48:21.537271 3163 raft_consensus.cc:493] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.537513 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 703441e1af8e4356afd22ef68dac87a2 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:21.537443 3163 raft_consensus.cc:515] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.538136 3163 leader_election.cc:290] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:21.538800 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1edc55caab904722bf19a1c71df601cf" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:21.538921 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d343821349d34633bfa9da8bc9dad031" } new_table_name: "default.Table_b668ef91cf2f47e7864a1fa580cda6d6" modify_external_catalogs: true
I20251024 19:48:21.539916 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1edc55caab904722bf19a1c71df601cf" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:21.540115 1505 raft_consensus.cc:2468] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:21.540453 1774 ts_tablet_manager.cc:1929] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.540583 1774 log.cc:1199] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/b240965605904533bd2af99f61aaef26
I20251024 19:48:21.540616 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 948) took 3260 ms. Trace:
W20251024 19:48:21.540587 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 66052.7us
I20251024 19:48:21.540571 1586 leader_election.cc:304] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:21.540699 1775 rpcz_store.cc:276] 1024 19:48:18.280066 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.286201 (+ 6135us) service_pool.cc:225] Handling call
1024 19:48:21.540468 (+3254267us) ts_tablet_manager.cc:1151] Deleting tablet 7a65fc59de5346d6a9344c5310455e53
1024 19:48:21.540481 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.540594 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3254011}
W20251024 19:48:21.541000 1586 leader_election.cc:343] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.541085 1774 ts_tablet_manager.cc:1950] T b240965605904533bd2af99f61aaef26 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:21.541371 3163 raft_consensus.cc:2804] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:21.541406 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 947) took 3265 ms. Trace:
I20251024 19:48:21.541446 3163 raft_consensus.cc:493] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:21.541500 3163 raft_consensus.cc:3060] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:21.541491 1775 rpcz_store.cc:276] 1024 19:48:18.276322 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.292008 (+ 15686us) service_pool.cc:225] Handling call
1024 19:48:21.541258 (+3249250us) ts_tablet_manager.cc:1151] Deleting tablet 81e3e2f8082d4c229e617f34f5eb5a78
1024 19:48:21.541269 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.541389 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3249047}
I20251024 19:48:21.541949 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 949) took 3248 ms. Trace:
I20251024 19:48:21.542032 1775 rpcz_store.cc:276] 1024 19:48:18.293079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.293129 (+ 50us) service_pool.cc:225] Handling call
1024 19:48:21.541859 (+3248730us) ts_tablet_manager.cc:1151] Deleting tablet f16ef86c5eb24489a66d6cd4bfa804fa
1024 19:48:21.541867 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.541935 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3248576}
I20251024 19:48:21.542356 1775 tablet_replica.cc:333] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.542541 1775 raft_consensus.cc:2243] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.542711 1775 raft_consensus.cc:2272] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:21.542819 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b6803d23a4714934b85190886611d8d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6803d23a4714934b85190886611d8d1 already in progress: creating tablet
W20251024 19:48:21.543144 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2c96c11697514af6bf4e80a67aadf748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c96c11697514af6bf4e80a67aadf748 already in progress: creating tablet
I20251024 19:48:21.543913 1775 ts_tablet_manager.cc:1916] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.544459 3163 raft_consensus.cc:515] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:21.544962 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b724eadbb86e4eeab4ba8dfd565290ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b724eadbb86e4eeab4ba8dfd565290ff already in progress: creating tablet
I20251024 19:48:21.545352 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 958) took 3498 ms. Trace:
I20251024 19:48:21.545430 1776 rpcz_store.cc:276] 1024 19:48:18.046607 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.046929 (+ 322us) service_pool.cc:225] Handling call
1024 19:48:21.305985 (+3259056us) ts_tablet_manager.cc:1151] Deleting tablet 844638a6b6944d0394df3e78a8ea51d6
1024 19:48:21.305998 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.332971 (+ 26973us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.472986 (+140015us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.545332 (+ 72346us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3258911}
I20251024 19:48:21.545661 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_cd3855acb8cc4a7486db5bbc7d37a26e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:21.545900 1776 tablet_replica.cc:333] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:21.546072 1776 raft_consensus.cc:2243] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:21.546366 1776 raft_consensus.cc:2272] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.546927 1776 ts_tablet_manager.cc:1916] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:21.547709 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet f4e812c43ade4831af50825474976c8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4e812c43ade4831af50825474976c8e already in progress: creating tablet
I20251024 19:48:21.548022 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_177f4c4e8d3b437aa14bb9e8a383510a" } modify_external_catalogs: true
I20251024 19:48:21.548748 3163 leader_election.cc:290] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:21.549531 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1edc55caab904722bf19a1c71df601cf" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:21.549787 1505 raft_consensus.cc:3060] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:21.552930 1505 raft_consensus.cc:2468] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:21.553416 1586 leader_election.cc:304] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:21.555166 1775 ts_tablet_manager.cc:1929] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251024 19:48:21.555174 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f10340e106b473eb715b74b129835b5 consensus state has no leader
I20251024 19:48:21.555286 1775 log.cc:1199] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/3e6db58fac5e4af9b27bde6c223aa845
I20251024 19:48:21.555472 3137 raft_consensus.cc:2804] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
W20251024 19:48:21.555552 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2671108031db449cae2135835fc5f739 consensus state has no leader
I20251024 19:48:21.555671 1775 ts_tablet_manager.cc:1950] T 3e6db58fac5e4af9b27bde6c223aa845 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:21.555684 3141 raft_consensus.cc:493] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.555807 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 844638a6b6944d0394df3e78a8ea51d6 (table default.Table_823526f1c3164a3ba70c44dd8388fb9c [id=fc5fe814531e4110ace3fb312aefa7ac]) successfully deleted
I20251024 19:48:21.555797 3141 raft_consensus.cc:515] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.556385 3141 leader_election.cc:290] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:21.556444 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 952) took 3157 ms. Trace:
I20251024 19:48:21.556665 3137 raft_consensus.cc:697] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:21.556914 3137 consensus_queue.cc:237] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.558352 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59956dad06064024ab171631ef6c1dca" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:21.558521 1505 raft_consensus.cc:2468] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:21.558898 1586 leader_election.cc:304] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:21.559110 3141 raft_consensus.cc:2804] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:21.559171 3141 raft_consensus.cc:493] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:21.559227 3141 raft_consensus.cc:3060] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:21.562168 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 703441e1af8e4356afd22ef68dac87a2
I20251024 19:48:21.562124 3141 raft_consensus.cc:515] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.562866 3141 leader_election.cc:290] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:21.563820 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59956dad06064024ab171631ef6c1dca" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:21.564004 1505 raft_consensus.cc:3060] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:21.564139 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 703441e1af8e4356afd22ef68dac87a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b119e8ceae04852964513ebfa7aa31f) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.564968 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_177f4c4e8d3b437aa14bb9e8a383510a" } modify_external_catalogs: true
I20251024 19:48:21.567243 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 936) took 1936 ms. Trace:
I20251024 19:48:21.567355 1774 rpcz_store.cc:276] 1024 19:48:19.631216 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.631278 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:21.297807 (+1666529us) ts_tablet_manager.cc:1151] Deleting tablet b240965605904533bd2af99f61aaef26
1024 19:48:21.299002 (+ 1195us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.514400 (+215398us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.540329 (+ 25929us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.567211 (+ 26882us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1666365}
I20251024 19:48:21.570081 1766 tablet_bootstrap.cc:492] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:21.570225 1766 ts_tablet_manager.cc:1403] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.149s user 0.013s sys 0.018s
I20251024 19:48:21.572144 1505 raft_consensus.cc:2468] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:21.574082 1586 leader_election.cc:304] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:21.574347 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 018d3e6aba6740f0bd82e8c83863bc4e consensus state has no leader
I20251024 19:48:21.574434 3163 raft_consensus.cc:2804] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:21.574630 3163 raft_consensus.cc:697] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
W20251024 19:48:21.574873 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet aeeaf256b1404653b690c712d9ef7a6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aeeaf256b1404653b690c712d9ef7a6d already in progress: creating tablet
I20251024 19:48:21.575244 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 50cea450eda9485daba40bf875a1b90c (table default.Table_ac4ce453efea4a268f70380f9ed99812 [id=bfd41da632b94dcf90b0e0d230dc8fb5]) successfully deleted
I20251024 19:48:21.574879 3163 consensus_queue.cc:237] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:21.576059 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet e7d6472ac7b046d89f56618fa4ed1811 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7d6472ac7b046d89f56618fa4ed1811 already in progress: creating tablet
W20251024 19:48:21.576367 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7a65fc59de5346d6a9344c5310455e53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a65fc59de5346d6a9344c5310455e53 already in progress: creating tablet
I20251024 19:48:21.577476 1774 tablet_replica.cc:333] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:21.577757 1774 raft_consensus.cc:2243] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.577939 1774 raft_consensus.cc:2272] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:21.577920 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 81e3e2f8082d4c229e617f34f5eb5a78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81e3e2f8082d4c229e617f34f5eb5a78 already in progress: creating tablet
W20251024 19:48:21.578231 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet f16ef86c5eb24489a66d6cd4bfa804fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f16ef86c5eb24489a66d6cd4bfa804fa already in progress: creating tablet
I20251024 19:48:21.581445 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet da7d38b63708427982591da1014aca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.582019 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 65b8d05d649244a58958a2d50265dc3a (DEFAULT_TABLE table=default.Table_7f7db12b2ef64686bb2a1b4d4cd7ed67 [id=812cc6b349d14e56bfcefc5347f1d615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.582396 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 5ba86757f4e84c679e515507dfc532c1 (DEFAULT_TABLE table=default.Table_be3f9c7f025449e6bef4abd44dc74fd6 [id=4a85ac78f7f142d78d421930d6957052]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.582597 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 703441e1af8e4356afd22ef68dac87a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b119e8ceae04852964513ebfa7aa31f) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.582779 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b with delete_type TABLET_DATA_DELETED (replaced by tablet 6bebcdb80476423b8e6c750f749a1771) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.583029 1766 raft_consensus.cc:359] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.583177 1766 raft_consensus.cc:385] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.583221 1766 raft_consensus.cc:740] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
W20251024 19:48:21.585958 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd3855acb8cc4a7486db5bbc7d37a26e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.587677 1776 ts_tablet_manager.cc:1929] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.587818 1776 log.cc:1199] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/6f630fd412ca4c4f8570359c1678e6af
I20251024 19:48:21.588044 1774 ts_tablet_manager.cc:1916] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.588196 1776 ts_tablet_manager.cc:1950] T 6f630fd412ca4c4f8570359c1678e6af P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:21.582401 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65b8d05d649244a58958a2d50265dc3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.583393 1766 consensus_queue.cc:260] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.590250 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba86757f4e84c679e515507dfc532c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.590732 1766 ts_tablet_manager.cc:1434] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20251024 19:48:21.591003 1766 tablet_bootstrap.cc:492] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:21.599030 1766 tablet_bootstrap.cc:654] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.588958 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 959) took 3490 ms. Trace:
I20251024 19:48:21.601735 1776 rpcz_store.cc:276] 1024 19:48:18.098323 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.101625 (+ 3302us) service_pool.cc:225] Handling call
1024 19:48:21.545851 (+3444226us) ts_tablet_manager.cc:1151] Deleting tablet 6f630fd412ca4c4f8570359c1678e6af
1024 19:48:21.545861 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.572581 (+ 26720us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.587524 (+ 14943us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.588936 (+ 1412us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3444022}
I20251024 19:48:21.602596 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 5ba86757f4e84c679e515507dfc532c1 (DEFAULT_TABLE table=default.Table_be3f9c7f025449e6bef4abd44dc74fd6 [id=4a85ac78f7f142d78d421930d6957052]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.603197 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba86757f4e84c679e515507dfc532c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.609241 1192 catalog_manager.cc:6719] ReplaceTablet: tablet 703441e1af8e4356afd22ef68dac87a2 deleted and replaced by tablet 2b119e8ceae04852964513ebfa7aa31f
I20251024 19:48:21.610204 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet a1fbbba88a294adfbf97c0343d86698e (table default.Table_7bc8cd4a930b44dfafa473297875594d [id=a60c8222aa0248c091fab8dc34100847]). Expected version 2 got 0
I20251024 19:48:21.556517 1775 rpcz_store.cc:276] 1024 19:48:18.398808 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.398855 (+ 47us) service_pool.cc:225] Handling call
1024 19:48:21.542315 (+3143460us) ts_tablet_manager.cc:1151] Deleting tablet 3e6db58fac5e4af9b27bde6c223aa845
1024 19:48:21.542323 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.551851 (+ 9528us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.555061 (+ 3210us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.556424 (+ 1363us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3143270}
I20251024 19:48:21.610807 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 956) took 3196 ms. Trace:
I20251024 19:48:21.610883 1775 rpcz_store.cc:276] 1024 19:48:18.414611 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.414691 (+ 80us) service_pool.cc:225] Handling call
1024 19:48:21.610669 (+3195978us) ts_tablet_manager.cc:1151] Deleting tablet 36ccb0cfba7a4a14b36d38897f4a9ef0
1024 19:48:21.610683 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.610792 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3195793}
I20251024 19:48:21.611209 1775 tablet_replica.cc:333] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.611382 1775 raft_consensus.cc:2243] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.611526 1775 raft_consensus.cc:2272] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.616381 1775 ts_tablet_manager.cc:1916] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.621723 1766 tablet_bootstrap.cc:492] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:21.623138 1766 ts_tablet_manager.cc:1403] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20251024 19:48:21.624434 1766 raft_consensus.cc:359] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.624626 1766 raft_consensus.cc:385] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.624699 1766 raft_consensus.cc:740] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.624929 1766 consensus_queue.cc:260] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.625468 1766 ts_tablet_manager.cc:1434] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:21.629940 1775 ts_tablet_manager.cc:1929] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:21.630051 1775 log.cc:1199] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/ee68243b4a664575b679950603bb250e
I20251024 19:48:21.630359 1775 ts_tablet_manager.cc:1950] T ee68243b4a664575b679950603bb250e P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:21.634187 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 68d85226ac5b4e4ab22a49dc419e375a
I20251024 19:48:21.636365 1766 tablet_bootstrap.cc:492] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:21.638690 1766 tablet_bootstrap.cc:654] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.647496 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 6f630fd412ca4c4f8570359c1678e6af (table default.Table_a5d11f93359546078ef5becf10539131 [id=e30a8747e9a6453e9567626b55ccfc1a]) successfully deleted
W20251024 19:48:21.647693 1176 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 45176.6us
I20251024 19:48:21.648850 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 04ae62c9bf6f44108d57d0760283d063 (DEFAULT_TABLE table=default.Table_2b311ba111854dabaeaaff96e18ee699 [id=fce0ef69ad3b433781be06d9c8a0149e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.648947 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 68d85226ac5b4e4ab22a49dc419e375a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.649055 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 186bac10f00d4396a3674a9037919e7c (DEFAULT_TABLE table=default.Table_ea78d03887214fd59d5ce1c5e137a7e0 [id=b4fdd246321146cdbf968d162389fe50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.648871 1630 tablet_service.cc:1505] Processing CreateTablet for tablet e9c0a61ce5f545a1b0263d4ce6bbb1c2 (DEFAULT_TABLE table=default.Table_8713def8ab7543788d6466f5d28125a2 [id=d513353177f34e2f8b911b6292805ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.648871 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 93d1bf52c34c49318de2ca4c27b35cf4 (DEFAULT_TABLE table=default.Table_ff301124b8ba431cb56766ff580f43ba [id=51d77585367949b598ff4444aeb36878]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.649289 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04ae62c9bf6f44108d57d0760283d063. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.649560 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d1bf52c34c49318de2ca4c27b35cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.653915 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 80688d7b8a394a2a910c4b8910fe4cf2 (DEFAULT_TABLE table=default.Table_4bb5f923e4074b398f16e0f53223fbf6 [id=0ea22d73321c47d1a445dbd3279214a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.654316 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80688d7b8a394a2a910c4b8910fe4cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.655000 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9c0a61ce5f545a1b0263d4ce6bbb1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.660060 2858 raft_consensus.cc:493] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.660219 2858 raft_consensus.cc:515] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.660923 2858 leader_election.cc:290] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:21.661669 1766 tablet_bootstrap.cc:492] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:21.661793 1766 ts_tablet_manager.cc:1403] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.004s
I20251024 19:48:21.661749 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee3c521876324808857f7724028c895f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:21.662287 1444 leader_election.cc:343] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.662590 1766 raft_consensus.cc:359] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.662762 1766 raft_consensus.cc:385] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.662807 1766 raft_consensus.cc:740] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.662986 1766 consensus_queue.cc:260] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.663426 1766 ts_tablet_manager.cc:1434] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:21.664184 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet b240965605904533bd2af99f61aaef26 (table default.Table_a1366192b3134e3abfc231729cb42344 [id=ae6b320a09934ac5873d3b35c33f2c65]) successfully deleted
W20251024 19:48:21.664420 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 70358us
I20251024 19:48:21.665758 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 186bac10f00d4396a3674a9037919e7c (DEFAULT_TABLE table=default.Table_ea78d03887214fd59d5ce1c5e137a7e0 [id=b4fdd246321146cdbf968d162389fe50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.666142 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 3e6db58fac5e4af9b27bde6c223aa845 (table default.Table_78e6806b2ede4763843892de1ab015e3 [id=1ac4b7fed3f145fc9ef5268266aeb9f1]) successfully deleted
I20251024 19:48:21.666143 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 80688d7b8a394a2a910c4b8910fe4cf2 (DEFAULT_TABLE table=default.Table_4bb5f923e4074b398f16e0f53223fbf6 [id=0ea22d73321c47d1a445dbd3279214a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.666231 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 68d85226ac5b4e4ab22a49dc419e375a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.666143 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 04ae62c9bf6f44108d57d0760283d063 (DEFAULT_TABLE table=default.Table_2b311ba111854dabaeaaff96e18ee699 [id=fce0ef69ad3b433781be06d9c8a0149e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.666483 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 93d1bf52c34c49318de2ca4c27b35cf4 (DEFAULT_TABLE table=default.Table_ff301124b8ba431cb56766ff580f43ba [id=51d77585367949b598ff4444aeb36878]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.666882 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 186bac10f00d4396a3674a9037919e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.666829 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ad2a772fc15e439bba0dfae4c45da5cf" } modify_external_catalogs: true
I20251024 19:48:21.666977 1475 tablet_service.cc:1505] Processing CreateTablet for tablet e9c0a61ce5f545a1b0263d4ce6bbb1c2 (DEFAULT_TABLE table=default.Table_8713def8ab7543788d6466f5d28125a2 [id=d513353177f34e2f8b911b6292805ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.667007 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ad2a772fc15e439bba0dfae4c45da5cf" } modify_external_catalogs: true
I20251024 19:48:21.668524 1766 tablet_bootstrap.cc:492] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:21.670168 1766 tablet_bootstrap.cc:654] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.674818 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 186bac10f00d4396a3674a9037919e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.675951 1189 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 945226cf57a54f7fa35d6b1291c657b3
I20251024 19:48:21.678236 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 957) took 3263 ms. Trace:
I20251024 19:48:21.678323 1775 rpcz_store.cc:276] 1024 19:48:18.414707 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.415054 (+ 347us) service_pool.cc:225] Handling call
1024 19:48:21.611172 (+3196118us) ts_tablet_manager.cc:1151] Deleting tablet ee68243b4a664575b679950603bb250e
1024 19:48:21.611179 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.619987 (+ 8808us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.629835 (+ 9848us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.678214 (+ 48379us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3195987}
I20251024 19:48:21.678638 1775 tablet_replica.cc:333] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.678839 1775 raft_consensus.cc:2243] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.678800 3163 consensus_queue.cc:1048] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:21.678987 1775 raft_consensus.cc:2272] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.680233 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9c0a61ce5f545a1b0263d4ce6bbb1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.681066 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d1bf52c34c49318de2ca4c27b35cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.681308 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04ae62c9bf6f44108d57d0760283d063. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.681962 1775 ts_tablet_manager.cc:1916] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.684005 3296 raft_consensus.cc:493] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.684190 3296 raft_consensus.cc:515] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.685046 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80688d7b8a394a2a910c4b8910fe4cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.691781 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 04e13290af90466d8766f571f72f20a4 (DEFAULT_TABLE table=default.Table_b057911324a844ac83247dcfdf437dcc [id=2755f6efc6974f6dadae7236abcfbbab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.692296 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e13290af90466d8766f571f72f20a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.693792 1621 tablet_service.cc:1505] Processing CreateTablet for tablet d72f4761820e430cb1a97d83cd30c75d (DEFAULT_TABLE table=default.Table_a99c55b564b946b6a76c2b981640cdcd [id=a9095592179b462396c9963617da5e7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.694177 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d72f4761820e430cb1a97d83cd30c75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.696044 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 2a1fe40804a24b968ba9c80c12ff8690 (DEFAULT_TABLE table=default.Table_a9322f2fceb94f43b0c6e920cc9da52c [id=52afc3cbd9014d06b6ebea7a6f5cf7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.696462 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1fe40804a24b968ba9c80c12ff8690. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.701215 2858 raft_consensus.cc:493] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.701463 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet 6424b2347ef642feb38a9596f2d2be1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.701407 2858 raft_consensus.cc:515] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.702267 2858 leader_election.cc:290] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:21.702332 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee3c521876324808857f7724028c895f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:21.702867 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 186bac10f00d4396a3674a9037919e7c (DEFAULT_TABLE table=default.Table_ea78d03887214fd59d5ce1c5e137a7e0 [id=b4fdd246321146cdbf968d162389fe50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.702965 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cff9bcc4924e0081518c284f074940" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:21.703063 1586 consensus_peers.cc:597] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:21.703253 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 80688d7b8a394a2a910c4b8910fe4cf2 (DEFAULT_TABLE table=default.Table_4bb5f923e4074b398f16e0f53223fbf6 [id=0ea22d73321c47d1a445dbd3279214a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.703358 3296 leader_election.cc:290] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:21.702867 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 5ba86757f4e84c679e515507dfc532c1 (DEFAULT_TABLE table=default.Table_be3f9c7f025449e6bef4abd44dc74fd6 [id=4a85ac78f7f142d78d421930d6957052]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.703567 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1edc55caab904722bf19a1c71df601cf" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:21.704049 3317 tablet.cc:1722] T d6de2c21796f4bc4904f88b654c3baec P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:21.707130 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet 129debb55f4a4b7295a8ad88e55ebc83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.709672 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 703441e1af8e4356afd22ef68dac87a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b119e8ceae04852964513ebfa7aa31f) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:21.710393 1444 leader_election.cc:343] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.712566 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 93d1bf52c34c49318de2ca4c27b35cf4 (DEFAULT_TABLE table=default.Table_ff301124b8ba431cb56766ff580f43ba [id=51d77585367949b598ff4444aeb36878]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.712680 1346 tablet_service.cc:1505] Processing CreateTablet for tablet e9c0a61ce5f545a1b0263d4ce6bbb1c2 (DEFAULT_TABLE table=default.Table_8713def8ab7543788d6466f5d28125a2 [id=d513353177f34e2f8b911b6292805ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.713066 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 2a1fe40804a24b968ba9c80c12ff8690 (DEFAULT_TABLE table=default.Table_a9322f2fceb94f43b0c6e920cc9da52c [id=52afc3cbd9014d06b6ebea7a6f5cf7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.713275 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 68d85226ac5b4e4ab22a49dc419e375a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:21.713636 1447 leader_election.cc:343] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.713781 1447 leader_election.cc:304] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.714039 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 04ae62c9bf6f44108d57d0760283d063 (DEFAULT_TABLE table=default.Table_2b311ba111854dabaeaaff96e18ee699 [id=fce0ef69ad3b433781be06d9c8a0149e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.714453 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129debb55f4a4b7295a8ad88e55ebc83" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:21.714516 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59956dad06064024ab171631ef6c1dca" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:21.715322 2858 raft_consensus.cc:2749] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.715620 1369 raft_consensus.cc:3060] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:21.718137 1774 ts_tablet_manager.cc:1929] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:21.718274 1774 log.cc:1199] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/c82f60819e7a442bbc0cc04659e14e87
I20251024 19:48:21.719213 2858 raft_consensus.cc:493] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.719352 2858 raft_consensus.cc:515] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.720518 2858 leader_election.cc:290] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:21.721395 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1fc0f99ca5c42bfb9663e21a7b761c9" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:21.721490 1775 ts_tablet_manager.cc:1929] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.721612 1775 log.cc:1199] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/3cf4596b333244cc9e9e35a128721e4a
I20251024 19:48:21.721977 1775 ts_tablet_manager.cc:1950] T 3cf4596b333244cc9e9e35a128721e4a P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
W20251024 19:48:21.722016 1444 leader_election.cc:343] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.722837 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 958) took 3308 ms. Trace:
I20251024 19:48:21.722613 1365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1fc0f99ca5c42bfb9663e21a7b761c9" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:21.726049 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_15b382636dbe430c991bff2369719fc7" } modify_external_catalogs: true
I20251024 19:48:21.726291 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 186bac10f00d4396a3674a9037919e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.726263 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_15b382636dbe430c991bff2369719fc7" } modify_external_catalogs: true
I20251024 19:48:21.727802 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_07aae1757a3444a4bbb8d729d6f08a1d" } modify_external_catalogs: true
I20251024 19:48:21.727950 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_07aae1757a3444a4bbb8d729d6f08a1d" } modify_external_catalogs: true
I20251024 19:48:21.722923 1775 rpcz_store.cc:276] 1024 19:48:18.414829 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.417000 (+ 2171us) service_pool.cc:225] Handling call
1024 19:48:21.678589 (+3261589us) ts_tablet_manager.cc:1151] Deleting tablet 3cf4596b333244cc9e9e35a128721e4a
1024 19:48:21.678599 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.684886 (+ 6287us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.721384 (+ 36498us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.722816 (+ 1432us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3261446}
I20251024 19:48:21.729993 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 962) took 3314 ms. Trace:
I20251024 19:48:21.730103 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80688d7b8a394a2a910c4b8910fe4cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.730094 1775 rpcz_store.cc:276] 1024 19:48:18.415170 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.421627 (+ 6457us) service_pool.cc:225] Handling call
1024 19:48:21.729818 (+3308191us) ts_tablet_manager.cc:1151] Deleting tablet b95989b98c77472faa7bf940cfa60432
1024 19:48:21.729831 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.729970 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3308025}
I20251024 19:48:21.730378 1775 tablet_replica.cc:333] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
W20251024 19:48:21.730926 1586 leader_election.cc:343] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.731247 1368 raft_consensus.cc:3060] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:21.731420 1586 leader_election.cc:343] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:21.731736 1586 consensus_peers.cc:597] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:21.731802 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba86757f4e84c679e515507dfc532c1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:21.733737 1586 consensus_peers.cc:597] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:21.733819 1447 leader_election.cc:343] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.734007 1447 leader_election.cc:304] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.734915 2858 raft_consensus.cc:2749] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.717784 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cff9bcc4924e0081518c284f074940" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:21.735776 1447 leader_election.cc:343] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.735889 1447 leader_election.cc:304] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.736420 2858 raft_consensus.cc:2749] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.717991 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59956dad06064024ab171631ef6c1dca" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:21.737396 1586 leader_election.cc:343] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.737958 3163 consensus_queue.cc:1048] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:21.741817 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d1bf52c34c49318de2ca4c27b35cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.741909 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4ddce416119e4f08b81be4b5e6760144
I20251024 19:48:21.743228 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129debb55f4a4b7295a8ad88e55ebc83" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:21.743736 1308 leader_election.cc:304] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.743804 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1fe40804a24b968ba9c80c12ff8690. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:21.744413 1586 consensus_peers.cc:597] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee68243b4a664575b679950603bb250e. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:21.745275 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9c0a61ce5f545a1b0263d4ce6bbb1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.745373 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ddce416119e4f08b81be4b5e6760144 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.745795 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04ae62c9bf6f44108d57d0760283d063. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.745949 1775 raft_consensus.cc:2243] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.746259 1775 pending_rounds.cc:70] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e: Trying to abort 1 pending ops.
I20251024 19:48:21.746301 1775 pending_rounds.cc:77] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429392004849664 op_type: NO_OP noop_request { }
I20251024 19:48:21.746377 1775 raft_consensus.cc:2889] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:21.746448 1775 raft_consensus.cc:2272] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.747284 1775 ts_tablet_manager.cc:1916] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.751986 2857 raft_consensus.cc:493] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.752115 2857 raft_consensus.cc:515] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:21.752477 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2671108031db449cae2135835fc5f739 (table default.Table_d3ffdf5cbfd84551b1e314d4a772be1a [id=2f6b7b685af84337bd64141ef2a9e992], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2671108031db449cae2135835fc5f739 consensus state has no leader
I20251024 19:48:21.752662 2857 leader_election.cc:290] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:21.752807 2857 raft_consensus.cc:493] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.752915 2857 raft_consensus.cc:515] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.753149 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23d4dd0b42d343cb9354ac698a8c761f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:21.753389 2857 leader_election.cc:290] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:21.753958 1444 leader_election.cc:343] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.754207 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee5356b5503c4425bf6648ade5f6330e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:21.754637 1444 leader_election.cc:343] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.755133 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23d4dd0b42d343cb9354ac698a8c761f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:21.755513 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet edb49884389c4dba957f0ee5af4ae0e4
W20251024 19:48:21.757659 1447 leader_election.cc:343] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.757877 1447 leader_election.cc:304] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.758402 2858 raft_consensus.cc:2749] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:21.758555 1586 consensus_peers.cc:597] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:21.758795 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee5356b5503c4425bf6648ade5f6330e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:21.759307 1447 leader_election.cc:343] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.759397 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6b41a041a50a4f2ca796ea21033f1ab9" } modify_external_catalogs: true
I20251024 19:48:21.759454 1447 leader_election.cc:304] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.759618 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6b41a041a50a4f2ca796ea21033f1ab9" } modify_external_catalogs: true
I20251024 19:48:21.759804 2858 raft_consensus.cc:2749] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.762707 3208 raft_consensus.cc:493] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.762856 3208 raft_consensus.cc:515] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.762894 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0fbe3a3cf6b9411495798ac85968de6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:21.763387 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fbe3a3cf6b9411495798ac85968de6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.763412 3208 leader_election.cc:290] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:21.764542 1308 leader_election.cc:343] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 3e6db58fac5e4af9b27bde6c223aa845
I20251024 19:48:21.764837 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6db58fac5e4af9b27bde6c223aa845" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:21.764912 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_423e7aaec4b54cb79a21504a67d8b143"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:21.765326 1305 leader_election.cc:304] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
W20251024 19:48:21.765379 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_423e7aaec4b54cb79a21504a67d8b143 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.765647 3208 raft_consensus.cc:2749] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.765688 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet edb49884389c4dba957f0ee5af4ae0e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.766705 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_391d10d0c6f44940bf592166d0b1aa40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:21.767109 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_391d10d0c6f44940bf592166d0b1aa40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.718590 1774 ts_tablet_manager.cc:1950] T c82f60819e7a442bbc0cc04659e14e87 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:21.769565 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 935) took 2139 ms. Trace:
I20251024 19:48:21.770866 1774 rpcz_store.cc:276] 1024 19:48:19.630507 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.632575 (+ 2068us) service_pool.cc:225] Handling call
1024 19:48:21.577405 (+1944830us) ts_tablet_manager.cc:1151] Deleting tablet c82f60819e7a442bbc0cc04659e14e87
1024 19:48:21.577420 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.708080 (+130660us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.718041 (+ 9961us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.769544 (+ 51503us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1944644}
I20251024 19:48:21.771319 1774 tablet_replica.cc:333] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:21.771514 1774 raft_consensus.cc:2243] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.771699 1774 raft_consensus.cc:2272] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:21.771724 1586 consensus_peers.cc:597] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb0405765c6146cbbb187af96a60be2e. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:21.772118 2923 consensus_queue.cc:1048] T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:21.772249 1774 ts_tablet_manager.cc:1916] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.775000 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d2f10524d22141078abbeeea11a129ef" } new_table_name: "default.Table_2065238ac26849bcb7f0c32cf69c1aac" modify_external_catalogs: true
I20251024 19:48:21.782951 1775 ts_tablet_manager.cc:1929] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.784200 1775 log.cc:1199] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/9a1031d785b646b095ece9be666d2a77
I20251024 19:48:21.784621 1775 ts_tablet_manager.cc:1950] T 9a1031d785b646b095ece9be666d2a77 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:21.784890 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 04e13290af90466d8766f571f72f20a4 (DEFAULT_TABLE table=default.Table_b057911324a844ac83247dcfdf437dcc [id=2755f6efc6974f6dadae7236abcfbbab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.788151 1341 tablet_service.cc:1505] Processing CreateTablet for tablet d72f4761820e430cb1a97d83cd30c75d (DEFAULT_TABLE table=default.Table_a99c55b564b946b6a76c2b981640cdcd [id=a9095592179b462396c9963617da5e7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.788642 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ddce416119e4f08b81be4b5e6760144 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.785892 1766 tablet_bootstrap.cc:492] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:21.789619 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet edb49884389c4dba957f0ee5af4ae0e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.789747 1766 ts_tablet_manager.cc:1403] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.121s user 0.005s sys 0.010s
I20251024 19:48:21.786319 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 2a1fe40804a24b968ba9c80c12ff8690 (DEFAULT_TABLE table=default.Table_a9322f2fceb94f43b0c6e920cc9da52c [id=52afc3cbd9014d06b6ebea7a6f5cf7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.786399 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 04e13290af90466d8766f571f72f20a4 (DEFAULT_TABLE table=default.Table_b057911324a844ac83247dcfdf437dcc [id=2755f6efc6974f6dadae7236abcfbbab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.790247 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e13290af90466d8766f571f72f20a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.790477 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 954) took 3376 ms. Trace:
I20251024 19:48:21.790570 1775 rpcz_store.cc:276] 1024 19:48:18.414426 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.434710 (+ 20284us) service_pool.cc:225] Handling call
1024 19:48:21.730334 (+3295624us) ts_tablet_manager.cc:1151] Deleting tablet 9a1031d785b646b095ece9be666d2a77
1024 19:48:21.730342 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.746177 (+ 15835us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x616000187c54. stack: 00007f26c12049f6 00007f26c12047b0 00007f26c01e03ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f26cd0ca100 00007f26ce417ae3 00007f26d0fd4287 00007f26d10072d4 00007f26d0fe7d70 00007f26d0fe7afd 0000000000a8867c 00007f26c124ad7f 00007f26c124e04b 00007f26c124dec8
1024 19:48:21.751310 (+ 5133us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.782843 (+ 31533us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.790457 (+ 7614us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35350400,"tablet-delete.queue_time_us":3295434}
I20251024 19:48:21.790892 1775 tablet_replica.cc:333] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.791098 1775 raft_consensus.cc:2243] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.791249 1775 raft_consensus.cc:2272] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.794189 1487 tablet_service.cc:1505] Processing CreateTablet for tablet d72f4761820e430cb1a97d83cd30c75d (DEFAULT_TABLE table=default.Table_a99c55b564b946b6a76c2b981640cdcd [id=a9095592179b462396c9963617da5e7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.794534 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d72f4761820e430cb1a97d83cd30c75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.794695 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 1b11c4f30236496e83f6336365cb5972 (DEFAULT_TABLE table=default.Table_686d12c4c37f4292aa2c15c0ee048f1e [id=25daed0639ff47bc9241e4870c98d825]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.795164 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b11c4f30236496e83f6336365cb5972. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.797114 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet edb49884389c4dba957f0ee5af4ae0e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.797699 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1fe40804a24b968ba9c80c12ff8690. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.797672 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ddce416119e4f08b81be4b5e6760144 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.790689 1766 raft_consensus.cc:359] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.799016 1766 raft_consensus.cc:385] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.799088 1766 raft_consensus.cc:740] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.799280 1766 consensus_queue.cc:260] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.799794 1766 ts_tablet_manager.cc:1434] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251024 19:48:21.800066 1766 tablet_bootstrap.cc:492] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:21.802415 2858 raft_consensus.cc:493] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.802717 2858 raft_consensus.cc:515] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:21.803328 2857 raft_consensus.cc:493] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.803432 2857 raft_consensus.cc:515] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.803850 2857 leader_election.cc:290] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:21.804388 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7be9290daabb404f955f05922d897fbf" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:21.804388 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd8aa1d9aca429dbfe46f472ebbf53b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:21.805109 1444 leader_election.cc:343] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.805109 2858 leader_election.cc:290] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:21.805481 1444 leader_election.cc:343] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.807636 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd8aa1d9aca429dbfe46f472ebbf53b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:21.808261 1447 leader_election.cc:343] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.808570 1447 leader_election.cc:304] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.809049 2858 raft_consensus.cc:2749] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.810205 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet c89fa8932dbb4f4fbf7fd9f57bff27c3
I20251024 19:48:21.811290 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet c89fa8932dbb4f4fbf7fd9f57bff27c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:21.811888 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7be9290daabb404f955f05922d897fbf" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:21.812338 1447 leader_election.cc:343] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.812539 1447 leader_election.cc:304] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.812958 2858 raft_consensus.cc:2749] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.816391 3208 raft_consensus.cc:493] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.816545 3208 raft_consensus.cc:515] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.817139 3208 leader_election.cc:290] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:21.817795 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d01572e51694d9bb58c3a59d4205102" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:21.818965 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d01572e51694d9bb58c3a59d4205102" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:21.819619 1305 leader_election.cc:304] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.819931 3208 raft_consensus.cc:2749] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.824369 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet c82f60819e7a442bbc0cc04659e14e87 (table default.Table_6f376570019346199a3c8c77d879fa1c [id=c5f7c2469f394bba8b91e2f20170963e]) successfully deleted
I20251024 19:48:21.824803 1768 tablet_bootstrap.cc:492] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:21.824946 1768 ts_tablet_manager.cc:1403] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.344s user 0.019s sys 0.027s
I20251024 19:48:21.824925 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet ee68243b4a664575b679950603bb250e (table default.Table_605117e2eb734e6b8dc34ceb4b5b5b71 [id=38b6e1273e1a43bd8b12604db5371677]) successfully deleted
I20251024 19:48:21.825518 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 3cf4596b333244cc9e9e35a128721e4a (table default.Table_25adc9559f2b43e2a4adc92dfc0c7807 [id=60d23d1c07704c33bc979f7ba025b102]) successfully deleted
W20251024 19:48:21.825767 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b95989b98c77472faa7bf940cfa60432 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b95989b98c77472faa7bf940cfa60432 already in progress: creating tablet
I20251024 19:48:21.825762 1768 raft_consensus.cc:359] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.825929 1768 raft_consensus.cc:385] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.825989 1768 raft_consensus.cc:740] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:21.826057 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 9a1031d785b646b095ece9be666d2a77 (table default.Table_a1d09325af844d71b290d95b8a4aaf12 [id=b62ec7bcad7c41b1b946e9ebf2f4bf39]) successfully deleted
I20251024 19:48:21.826282 1767 tablet_bootstrap.cc:654] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.826228 1768 consensus_queue.cc:260] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.826743 1768 ts_tablet_manager.cc:1434] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:21.827026 1768 tablet_bootstrap.cc:492] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:21.830360 1775 ts_tablet_manager.cc:1916] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.831833 1185 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 067631b128fd4a1e9ab72c9507dab6c4 (table default.Table_814a6dad46d54945b70e652b7e0f0032 [id=3f394c59d82b44928beb22f42312a4a3]). Expected version 1 got 0
I20251024 19:48:21.832186 1185 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 2d01572e51694d9bb58c3a59d4205102 (table default.Table_bf15d3c700774490b1860d3d8865a95e [id=2ea13e78137d47429d7cbe374d2113ef]). Expected version 1 got 0
I20251024 19:48:21.838691 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 28f93be6762146398421ac19d2e84e0c (DEFAULT_TABLE table=default.Table_9ba52b13507546c4b00c9109363aacc9 [id=1c9e66f269594f5bb23f747ad970de72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.839208 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f93be6762146398421ac19d2e84e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.841768 2858 raft_consensus.cc:493] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.841940 2858 raft_consensus.cc:515] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.842684 2858 leader_election.cc:290] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:21.842999 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30bce965315d43948ecb8282c829c557" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:21.843256 1364 raft_consensus.cc:3060] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:21.602701 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 961) took 3454 ms. Trace:
I20251024 19:48:21.843565 1776 rpcz_store.cc:276] 1024 19:48:18.148466 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.153220 (+ 4754us) service_pool.cc:225] Handling call
1024 19:48:21.602566 (+3449346us) ts_tablet_manager.cc:1151] Deleting tablet e7d6472ac7b046d89f56618fa4ed1811
1024 19:48:21.602578 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.602683 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3449092}
W20251024 19:48:21.843686 1444 leader_election.cc:343] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.844264 3296 raft_consensus.cc:2749] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.844305 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30bce965315d43948ecb8282c829c557" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:21.844794 1447 leader_election.cc:343] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.844947 1447 leader_election.cc:304] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.845361 2858 raft_consensus.cc:2749] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.855012 1183 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_1a079b981b694617b3baf9897dfb37ae [id=18ec057e920e4c21ae8cbe298058b6f3] alter complete (version 2)
I20251024 19:48:21.856668 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 1b11c4f30236496e83f6336365cb5972 (DEFAULT_TABLE table=default.Table_686d12c4c37f4292aa2c15c0ee048f1e [id=25daed0639ff47bc9241e4870c98d825]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:21.860112 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2fe735d85414bcda79ea4399adc041f consensus state has no leader
I20251024 19:48:21.863874 1775 ts_tablet_manager.cc:1929] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.867553 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_31e350af7b2a4e38981c3db02a346e52" } new_table_name: "default.Table_1037786a8dff4c57b157b251d8f39cfb" modify_external_catalogs: true
W20251024 19:48:21.878443 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7074697285141e3a62f8f372d86a3c9 consensus state has no leader
I20251024 19:48:21.878533 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet c89fa8932dbb4f4fbf7fd9f57bff27c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.881346 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d989d66502a645bc8cc2c0cef2bc0d35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:21.865032 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet c89fa8932dbb4f4fbf7fd9f57bff27c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:21.882110 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d989d66502a645bc8cc2c0cef2bc0d35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.882227 1775 log.cc:1199] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/6424b2347ef642feb38a9596f2d2be1d
I20251024 19:48:21.880393 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_35b260ec41dd461eb1b08fd8485c69c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:21.885617 1775 ts_tablet_manager.cc:1950] T 6424b2347ef642feb38a9596f2d2be1d P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
W20251024 19:48:21.886363 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35b260ec41dd461eb1b08fd8485c69c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:21.854838 1586 consensus_peers.cc:597] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:21.890560 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet e7d6472ac7b046d89f56618fa4ed1811 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7d6472ac7b046d89f56618fa4ed1811 already in progress: creating tablet
I20251024 19:48:21.890918 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 964) took 3612 ms. Trace:
I20251024 19:48:21.891166 2858 raft_consensus.cc:493] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.891117 1776 rpcz_store.cc:276] 1024 19:48:18.278779 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.286826 (+ 8047us) service_pool.cc:225] Handling call
1024 19:48:21.890635 (+3603809us) ts_tablet_manager.cc:1151] Deleting tablet 7a65fc59de5346d6a9344c5310455e53
1024 19:48:21.890650 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.890862 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3602519}
W20251024 19:48:21.891757 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7a65fc59de5346d6a9344c5310455e53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a65fc59de5346d6a9344c5310455e53 already in progress: creating tablet
I20251024 19:48:21.855304 2923 raft_consensus.cc:493] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.892050 2923 raft_consensus.cc:515] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.892107 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 963) took 3618 ms. Trace:
I20251024 19:48:21.892375 1776 rpcz_store.cc:276] 1024 19:48:18.273990 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.289138 (+ 15148us) service_pool.cc:225] Handling call
1024 19:48:21.891849 (+3602711us) ts_tablet_manager.cc:1151] Deleting tablet 81e3e2f8082d4c229e617f34f5eb5a78
1024 19:48:21.891861 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.892057 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3602463}
I20251024 19:48:21.892748 2923 leader_election.cc:290] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:21.893064 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 81e3e2f8082d4c229e617f34f5eb5a78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81e3e2f8082d4c229e617f34f5eb5a78 already in progress: creating tablet
I20251024 19:48:21.893457 2858 raft_consensus.cc:515] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.893677 1766 tablet_bootstrap.cc:654] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.866788 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 1b11c4f30236496e83f6336365cb5972 (DEFAULT_TABLE table=default.Table_686d12c4c37f4292aa2c15c0ee048f1e [id=25daed0639ff47bc9241e4870c98d825]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.894204 2858 leader_election.cc:290] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:21.894901 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1fbbba88a294adfbf97c0343d86698e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:21.895285 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73278db2097d4001a538d62dacb97c9b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:21.895597 1586 leader_election.cc:343] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.895560 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 955) took 3476 ms. Trace:
W20251024 19:48:21.896086 1447 leader_election.cc:343] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.896136 1775 rpcz_store.cc:276] 1024 19:48:18.414534 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.442116 (+ 27582us) service_pool.cc:225] Handling call
1024 19:48:21.790849 (+3348733us) ts_tablet_manager.cc:1151] Deleting tablet 6424b2347ef642feb38a9596f2d2be1d
1024 19:48:21.790857 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.856348 (+ 65491us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.863755 (+ 7407us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.890867 (+ 27112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3348527}
I20251024 19:48:21.898181 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1fbbba88a294adfbf97c0343d86698e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:21.898406 1504 raft_consensus.cc:2468] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:21.899156 1586 leader_election.cc:304] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:21.899106 3137 consensus_queue.cc:1048] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:21.899598 3137 raft_consensus.cc:2804] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:21.899730 3137 raft_consensus.cc:493] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:21.899881 3137 raft_consensus.cc:3060] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:21.901456 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b11c4f30236496e83f6336365cb5972. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.902770 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7c6e82a967ab492ebf7d644969562282"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:21.903484 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c6e82a967ab492ebf7d644969562282 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.903777 3137 raft_consensus.cc:515] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.904635 3137 leader_election.cc:290] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:21.905107 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d72f4761820e430cb1a97d83cd30c75d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:21.905229 3330 log.cc:927] Time spent T 1c92b080805845c6af8f1cb160110915 P af3952de050140d383f6b0de7666fa97: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20251024 19:48:21.905854 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1fbbba88a294adfbf97c0343d86698e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:21.906371 1586 leader_election.cc:343] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.909694 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b11c4f30236496e83f6336365cb5972. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.909969 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04e13290af90466d8766f571f72f20a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.920135 1767 tablet_bootstrap.cc:492] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:21.920353 1767 ts_tablet_manager.cc:1403] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.430s user 0.001s sys 0.005s
I20251024 19:48:21.921998 1767 raft_consensus.cc:359] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.922255 1767 raft_consensus.cc:385] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:21.922394 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_de48e6da258b4f55b0167897017342fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:21.923002 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de48e6da258b4f55b0167897017342fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.923970 1774 ts_tablet_manager.cc:1929] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:21.924095 1774 log.cc:1199] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/6f630fd412ca4c4f8570359c1678e6af
I20251024 19:48:21.924293 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f76a1631002d4ee0a3323bcbd2e339d3" } new_table_name: "default.Table_562481ab77914c9da01e8f0c63e93035" modify_external_catalogs: true
I20251024 19:48:21.924465 1774 ts_tablet_manager.cc:1950] T 6f630fd412ca4c4f8570359c1678e6af P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:21.924793 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 934) took 2294 ms. Trace:
I20251024 19:48:21.925776 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_075cb16ace064e7a8840d0d6d2ede7ec" } new_table_name: "default.Table_0929cd0dc2cb455ca05d93504fb52ca8" modify_external_catalogs: true
I20251024 19:48:21.905121 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1fbbba88a294adfbf97c0343d86698e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:21.926990 1505 raft_consensus.cc:3060] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:21.930471 1505 raft_consensus.cc:2468] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:21.931020 1586 leader_election.cc:304] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:21.933408 2923 raft_consensus.cc:2804] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:21.933544 2923 raft_consensus.cc:697] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:21.933862 2923 consensus_queue.cc:237] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.934343 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73278db2097d4001a538d62dacb97c9b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:21.934405 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 965) took 3641 ms. Trace:
I20251024 19:48:21.934487 1776 rpcz_store.cc:276] 1024 19:48:18.293231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.300345 (+ 7114us) service_pool.cc:225] Handling call
1024 19:48:21.893170 (+3592825us) ts_tablet_manager.cc:1151] Deleting tablet f16ef86c5eb24489a66d6cd4bfa804fa
1024 19:48:21.934227 (+ 41057us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.934382 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3592600}
W20251024 19:48:21.934921 1444 leader_election.cc:343] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:21.934934 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f16ef86c5eb24489a66d6cd4bfa804fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f16ef86c5eb24489a66d6cd4bfa804fa already in progress: creating tablet
I20251024 19:48:21.935041 1444 leader_election.cc:304] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.936388 2857 raft_consensus.cc:2749] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.948071 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 968) took 3548 ms. Trace:
I20251024 19:48:21.948190 1776 rpcz_store.cc:276] 1024 19:48:18.399878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.399940 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:21.935189 (+3535249us) ts_tablet_manager.cc:1151] Deleting tablet 36ccb0cfba7a4a14b36d38897f4a9ef0
1024 19:48:21.947639 (+ 12450us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.948044 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3406513}
I20251024 19:48:21.948925 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 973) took 3543 ms. Trace:
I20251024 19:48:21.949013 1776 rpcz_store.cc:276] 1024 19:48:18.405221 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.405799 (+ 578us) service_pool.cc:225] Handling call
1024 19:48:21.948813 (+3543014us) ts_tablet_manager.cc:1151] Deleting tablet b95989b98c77472faa7bf940cfa60432
1024 19:48:21.948823 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.948905 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3403095}
W20251024 19:48:21.949380 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b95989b98c77472faa7bf940cfa60432 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b95989b98c77472faa7bf940cfa60432 already in progress: creating tablet
I20251024 19:48:21.908861 1775 tablet_replica.cc:333] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:21.912788 2858 raft_consensus.cc:493] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:21.949868 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 988) took 3178 ms. Trace:
I20251024 19:48:21.949863 2858 raft_consensus.cc:515] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:21.950116 1776 rpcz_store.cc:276] 1024 19:48:18.771099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.771166 (+ 67us) service_pool.cc:225] Handling call
1024 19:48:21.949662 (+3178496us) ts_tablet_manager.cc:1151] Deleting tablet 54bc7c7764584e11948d2831a4e5e0b6
1024 19:48:21.949671 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.949751 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3178049}
I20251024 19:48:21.950549 2858 leader_election.cc:290] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:21.950594 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 989) took 3154 ms. Trace:
W20251024 19:48:21.950569 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 54bc7c7764584e11948d2831a4e5e0b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54bc7c7764584e11948d2831a4e5e0b6 already in progress: creating tablet
I20251024 19:48:21.950723 1776 rpcz_store.cc:276] 1024 19:48:18.795872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.795926 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:21.950350 (+3154424us) ts_tablet_manager.cc:1151] Deleting tablet de2ebb79d4fe4a1bbac372bb26f370ed
1024 19:48:21.950359 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.950429 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3149785}
I20251024 19:48:21.951424 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 990) took 3116 ms. Trace:
W20251024 19:48:21.951344 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet de2ebb79d4fe4a1bbac372bb26f370ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de2ebb79d4fe4a1bbac372bb26f370ed already in progress: creating tablet
I20251024 19:48:21.951592 1776 rpcz_store.cc:276] 1024 19:48:18.834965 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.835031 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:21.951147 (+3116116us) ts_tablet_manager.cc:1151] Deleting tablet c9761b71a7294c78bccb5be86529dbe8
1024 19:48:21.951158 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.951249 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3115914}
I20251024 19:48:21.951861 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1d26e02ad2945b39cf27738ed5b7cf4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:21.952411 1447 leader_election.cc:343] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.952870 1768 tablet_bootstrap.cc:654] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.953729 1775 raft_consensus.cc:2243] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.953989 1775 raft_consensus.cc:2272] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.954692 1775 ts_tablet_manager.cc:1916] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.965204 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 28f93be6762146398421ac19d2e84e0c (DEFAULT_TABLE table=default.Table_9ba52b13507546c4b00c9109363aacc9 [id=1c9e66f269594f5bb23f747ad970de72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.965462 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 28f93be6762146398421ac19d2e84e0c (DEFAULT_TABLE table=default.Table_9ba52b13507546c4b00c9109363aacc9 [id=1c9e66f269594f5bb23f747ad970de72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:21.965711 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f93be6762146398421ac19d2e84e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.924861 1774 rpcz_store.cc:276] 1024 19:48:19.630381 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.632785 (+ 2404us) service_pool.cc:225] Handling call
1024 19:48:21.771273 (+2138488us) ts_tablet_manager.cc:1151] Deleting tablet 6f630fd412ca4c4f8570359c1678e6af
1024 19:48:21.771282 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.882644 (+111362us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.923881 (+ 41237us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:21.924778 (+ 897us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2138229}
I20251024 19:48:21.967003 1774 tablet_replica.cc:333] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:21.967271 1774 raft_consensus.cc:2243] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:21.967495 1774 raft_consensus.cc:2272] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.969396 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf4596b333244cc9e9e35a128721e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.969946 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet ee3c521876324808857f7724028c895f with delete_type TABLET_DATA_DELETED (replaced by tablet 94b35252e63f4a8ba305cc029ddb5ae8) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:21.951769 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1d26e02ad2945b39cf27738ed5b7cf4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:21.922351 1767 raft_consensus.cc:740] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
W20251024 19:48:21.973197 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet c9761b71a7294c78bccb5be86529dbe8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9761b71a7294c78bccb5be86529dbe8 already in progress: creating tablet
W20251024 19:48:21.973356 1444 leader_election.cc:343] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:21.973475 1444 leader_election.cc:304] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:21.973376 1776 tablet_replica.cc:333] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:21.973491 1767 consensus_queue.cc:260] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:21.973901 1776 raft_consensus.cc:2243] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:21.973963 2858 raft_consensus.cc:2749] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:21.974151 1767 ts_tablet_manager.cc:1434] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.054s user 0.002s sys 0.000s
I20251024 19:48:21.972594 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 6f630fd412ca4c4f8570359c1678e6af (table default.Table_a5d11f93359546078ef5becf10539131 [id=e30a8747e9a6453e9567626b55ccfc1a]) successfully deleted
I20251024 19:48:21.974506 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f93be6762146398421ac19d2e84e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:21.974532 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 6424b2347ef642feb38a9596f2d2be1d (table default.Table_da1c2655e0de479e91cf0a21e1a728b5 [id=42ffad851c714cb28e672b448efb33bd]) successfully deleted
I20251024 19:48:21.976469 1767 tablet_bootstrap.cc:492] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:21.979410 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet ee68243b4a664575b679950603bb250e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.979700 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e6db58fac5e4af9b27bde6c223aa845 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:21.980186 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6b101c11690c47319e38598ef667655b" } new_table_name: "default.Table_2979f4edd66d4bbabe8489c71bc04bf8" modify_external_catalogs: true
I20251024 19:48:21.980291 1767 tablet_bootstrap.cc:654] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:21.981738 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9108635a402d40dba89df9a92ae028b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:21.982329 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9108635a402d40dba89df9a92ae028b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:21.968262 1774 ts_tablet_manager.cc:1916] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.988837 1186 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 2d01572e51694d9bb58c3a59d4205102 (table default.Table_bf15d3c700774490b1860d3d8865a95e [id=2ea13e78137d47429d7cbe374d2113ef]). Expected version 1 got 0
I20251024 19:48:21.990015 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b33dc95780534701ab219c111a03ee38" } modify_external_catalogs: true
I20251024 19:48:21.990186 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b33dc95780534701ab219c111a03ee38" } modify_external_catalogs: true
I20251024 19:48:21.993116 1187 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet a1fbbba88a294adfbf97c0343d86698e (table default.Table_7bc8cd4a930b44dfafa473297875594d [id=a60c8222aa0248c091fab8dc34100847]). Expected version 2 got 0
W20251024 19:48:21.994002 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef837960a0544202834a0ca509089b29 consensus state has no leader
I20251024 19:48:21.994902 1776 raft_consensus.cc:2272] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:21.995486 1776 ts_tablet_manager.cc:1916] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:21.995671 1192 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 2d01572e51694d9bb58c3a59d4205102 (table default.Table_bf15d3c700774490b1860d3d8865a95e [id=2ea13e78137d47429d7cbe374d2113ef]). Expected version 1 got 0
I20251024 19:48:22.001358 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3a4994ec76cc4afaa6e9bb2a0fa8a353"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.002455 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a4994ec76cc4afaa6e9bb2a0fa8a353 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.005723 1191 catalog_manager.cc:5649] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:22.006309 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet a1fbbba88a294adfbf97c0343d86698e (table default.Table_7bc8cd4a930b44dfafa473297875594d [id=a60c8222aa0248c091fab8dc34100847]). Expected version 2 got 0
I20251024 19:48:22.013046 1768 tablet_bootstrap.cc:492] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:22.013211 1768 ts_tablet_manager.cc:1403] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.186s user 0.002s sys 0.003s
I20251024 19:48:22.013962 1768 raft_consensus.cc:359] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:22.014135 1768 raft_consensus.cc:385] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.014184 1768 raft_consensus.cc:740] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.014420 1768 consensus_queue.cc:260] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:22.014932 1768 ts_tablet_manager.cc:1434] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251024 19:48:22.020959 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 027e12c7961247a4a1664b85e4ddae4c consensus state has no leader
W20251024 19:48:22.021492 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 08b1b461a5de434d83ba5b6c4587a66f consensus state has no leader
I20251024 19:48:22.025103 1364 raft_consensus.cc:3060] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:22.026424 1768 tablet_bootstrap.cc:492] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:22.030547 1767 tablet_bootstrap.cc:492] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:22.030786 1767 ts_tablet_manager.cc:1403] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.001s
I20251024 19:48:22.031633 1767 raft_consensus.cc:359] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.031895 1767 raft_consensus.cc:385] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.032008 1767 raft_consensus.cc:740] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.032306 1767 consensus_queue.cc:260] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.033080 1767 ts_tablet_manager.cc:1434] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:22.034169 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_00d39085c7a9429eba70f98bba9edd11" } modify_external_catalogs: true
I20251024 19:48:22.034337 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_00d39085c7a9429eba70f98bba9edd11" } modify_external_catalogs: true
I20251024 19:48:22.046839 1768 tablet_bootstrap.cc:654] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.051311 1767 tablet_bootstrap.cc:492] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:22.053074 1767 tablet_bootstrap.cc:654] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.053735 1774 ts_tablet_manager.cc:1929] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:22.061610 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1ff2aa8257594c2fa59a715522aea99d" } modify_external_catalogs: true
I20251024 19:48:22.061873 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1ff2aa8257594c2fa59a715522aea99d" } modify_external_catalogs: true
I20251024 19:48:22.063228 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ac91d48ce11a4e0cbed30752c680d6eb" } modify_external_catalogs: true
I20251024 19:48:22.063432 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ac91d48ce11a4e0cbed30752c680d6eb" } modify_external_catalogs: true
I20251024 19:48:22.063465 1776 ts_tablet_manager.cc:1929] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.063563 1776 log.cc:1199] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/cb0405765c6146cbbb187af96a60be2e
I20251024 19:48:22.063933 1776 ts_tablet_manager.cc:1950] T cb0405765c6146cbbb187af96a60be2e P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:22.076274 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet aa08555dbe1e4ad8968c534510eca9ff
I20251024 19:48:22.077198 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet aa08555dbe1e4ad8968c534510eca9ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:22.090544 1767 tablet_bootstrap.cc:492] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:22.090847 1767 ts_tablet_manager.cc:1403] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20251024 19:48:22.091781 1767 raft_consensus.cc:359] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:22.092046 1767 raft_consensus.cc:385] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.092147 1767 raft_consensus.cc:740] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.092442 1767 consensus_queue.cc:260] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:22.093091 1767 ts_tablet_manager.cc:1434] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:22.093418 1767 tablet_bootstrap.cc:492] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:22.094117 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3e8d42d7a60a43bf99d216729f8533ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.094780 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e8d42d7a60a43bf99d216729f8533ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.095229 1767 tablet_bootstrap.cc:654] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.096555 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b94c179f1c064b5ab2010fe812734297"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.097106 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b94c179f1c064b5ab2010fe812734297 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.098549 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5899d327ab3844daa60d50f0035a4751"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.099062 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5899d327ab3844daa60d50f0035a4751 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.115994 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_af63b6be7a7241feb3512fe1856876bc" } modify_external_catalogs: true
I20251024 19:48:22.116178 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_af63b6be7a7241feb3512fe1856876bc" } modify_external_catalogs: true
I20251024 19:48:22.122908 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet aa08555dbe1e4ad8968c534510eca9ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:22.126261 1766 tablet_bootstrap.cc:492] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:22.126430 1766 ts_tablet_manager.cc:1403] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.326s user 0.004s sys 0.008s
I20251024 19:48:22.127355 1766 raft_consensus.cc:359] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:22.127537 1766 raft_consensus.cc:385] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.127617 1766 raft_consensus.cc:740] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.127872 1766 consensus_queue.cc:260] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:22.128381 1766 ts_tablet_manager.cc:1434] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:22.128680 1766 tablet_bootstrap.cc:492] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:22.130298 1766 tablet_bootstrap.cc:654] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.074366 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf4596b333244cc9e9e35a128721e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.125029 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet aa08555dbe1e4ad8968c534510eca9ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.142344 1775 ts_tablet_manager.cc:1929] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.142525 1775 log.cc:1199] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/129debb55f4a4b7295a8ad88e55ebc83
I20251024 19:48:22.142974 1775 ts_tablet_manager.cc:1950] T 129debb55f4a4b7295a8ad88e55ebc83 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:22.143787 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 953) took 3729 ms. Trace:
I20251024 19:48:22.143932 1767 tablet_bootstrap.cc:492] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:22.143893 1775 rpcz_store.cc:276] 1024 19:48:18.414309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.461796 (+ 47487us) service_pool.cc:225] Handling call
1024 19:48:21.897927 (+3436131us) ts_tablet_manager.cc:1151] Deleting tablet 129debb55f4a4b7295a8ad88e55ebc83
1024 19:48:21.908545 (+ 10618us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.959471 (+ 50926us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.142201 (+182730us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.143743 (+ 1542us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1858,"tablet-delete.queue_time_us":3435764}
I20251024 19:48:22.144095 1767 ts_tablet_manager.cc:1403] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20251024 19:48:22.144475 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 129debb55f4a4b7295a8ad88e55ebc83 (table default.Table_c777799df90e491992909e8a45205a06 [id=d11ca7b446a44dd28c9c243a6d7787e6]) successfully deleted
I20251024 19:48:22.144879 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 978) took 3249 ms. Trace:
I20251024 19:48:22.144901 1767 raft_consensus.cc:359] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.145092 1767 raft_consensus.cc:385] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.145160 1767 raft_consensus.cc:740] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.145275 1775 rpcz_store.cc:276] 1024 19:48:18.895345 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.895458 (+ 113us) service_pool.cc:225] Handling call
1024 19:48:22.144732 (+3249274us) ts_tablet_manager.cc:1151] Deleting tablet de2ebb79d4fe4a1bbac372bb26f370ed
1024 19:48:22.144742 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.144838 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3248915}
I20251024 19:48:22.145375 1767 consensus_queue.cc:260] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:22.145916 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet de2ebb79d4fe4a1bbac372bb26f370ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de2ebb79d4fe4a1bbac372bb26f370ed already in progress: creating tablet
I20251024 19:48:22.147410 1767 ts_tablet_manager.cc:1434] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:22.149009 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 977) took 3254 ms. Trace:
I20251024 19:48:22.149267 1775 rpcz_store.cc:276] 1024 19:48:18.894153 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.901103 (+ 6950us) service_pool.cc:225] Handling call
1024 19:48:22.146912 (+3245809us) ts_tablet_manager.cc:1151] Deleting tablet 54bc7c7764584e11948d2831a4e5e0b6
1024 19:48:22.146924 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.148898 (+ 1974us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3243178}
W20251024 19:48:22.149879 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 54bc7c7764584e11948d2831a4e5e0b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54bc7c7764584e11948d2831a4e5e0b6 already in progress: creating tablet
I20251024 19:48:22.150483 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 979) took 3240 ms. Trace:
I20251024 19:48:22.150722 1775 rpcz_store.cc:276] 1024 19:48:18.909947 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.910037 (+ 90us) service_pool.cc:225] Handling call
1024 19:48:22.150246 (+3240209us) ts_tablet_manager.cc:1151] Deleting tablet c9761b71a7294c78bccb5be86529dbe8
1024 19:48:22.150257 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.150359 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3239954}
W20251024 19:48:22.151296 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet c9761b71a7294c78bccb5be86529dbe8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9761b71a7294c78bccb5be86529dbe8 already in progress: creating tablet
I20251024 19:48:22.151903 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 981) took 2841 ms. Trace:
I20251024 19:48:22.152029 1774 log.cc:1199] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/639836c656f44a819036f4d00e055bd9
I20251024 19:48:22.152102 1775 rpcz_store.cc:276] 1024 19:48:19.310680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.310753 (+ 73us) service_pool.cc:225] Handling call
1024 19:48:22.151665 (+2840912us) ts_tablet_manager.cc:1151] Deleting tablet 346cb0bb8a0d43f7a15b788a303a6bcf
1024 19:48:22.151678 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.151771 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2840686}
I20251024 19:48:22.152518 1774 ts_tablet_manager.cc:1950] T 639836c656f44a819036f4d00e055bd9 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:48:22.152573 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 346cb0bb8a0d43f7a15b788a303a6bcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 346cb0bb8a0d43f7a15b788a303a6bcf already in progress: creating tablet
I20251024 19:48:22.152925 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 933) took 2523 ms. Trace:
I20251024 19:48:22.153038 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 982) took 2834 ms. Trace:
I20251024 19:48:22.153055 1774 rpcz_store.cc:276] 1024 19:48:19.629605 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.633091 (+ 3486us) service_pool.cc:225] Handling call
1024 19:48:21.966934 (+2333843us) ts_tablet_manager.cc:1151] Deleting tablet 639836c656f44a819036f4d00e055bd9
1024 19:48:21.966944 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:21.989454 (+ 22510us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.053600 (+ 64146us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.152907 (+ 99307us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2333683}
I20251024 19:48:22.153223 1775 rpcz_store.cc:276] 1024 19:48:19.318096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.318165 (+ 69us) service_pool.cc:225] Handling call
1024 19:48:22.152848 (+2834683us) ts_tablet_manager.cc:1151] Deleting tablet af918843c74940888c709d9a05aa6e0f
1024 19:48:22.152856 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.152929 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2834084}
I20251024 19:48:22.153513 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 639836c656f44a819036f4d00e055bd9 (table default.Table_6eb10ae51f3a43c2ab15f362708fba94 [id=f3080df28ee94876bb6b790cea45a4e8]) successfully deleted
W20251024 19:48:22.153985 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet af918843c74940888c709d9a05aa6e0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af918843c74940888c709d9a05aa6e0f already in progress: creating tablet
I20251024 19:48:22.154472 1768 tablet_bootstrap.cc:492] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:22.154458 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1005) took 2448 ms. Trace:
I20251024 19:48:22.154675 1768 ts_tablet_manager.cc:1403] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.128s user 0.001s sys 0.005s
I20251024 19:48:22.154611 1775 rpcz_store.cc:276] 1024 19:48:19.706376 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.706446 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:22.154261 (+2447815us) ts_tablet_manager.cc:1151] Deleting tablet e38ef2ac19d14f9c91fa8abe86cc7987
1024 19:48:22.154271 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.154341 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2447537}
I20251024 19:48:22.155352 1775 tablet_replica.cc:333] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:22.155794 1775 raft_consensus.cc:2243] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.155813 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 937) took 2364 ms. Trace:
I20251024 19:48:22.155884 1774 rpcz_store.cc:276] 1024 19:48:19.791156 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.792722 (+ 1566us) service_pool.cc:225] Handling call
1024 19:48:22.153559 (+2360837us) ts_tablet_manager.cc:1151] Deleting tablet e38ef2ac19d14f9c91fa8abe86cc7987
1024 19:48:22.153568 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.155795 (+ 2227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2360324}
I20251024 19:48:22.156381 1775 raft_consensus.cc:2272] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.156358 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 938) took 2311 ms. Trace:
I20251024 19:48:22.156561 1774 rpcz_store.cc:276] 1024 19:48:19.844804 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.844881 (+ 77us) service_pool.cc:225] Handling call
1024 19:48:22.156251 (+2311370us) ts_tablet_manager.cc:1151] Deleting tablet 9a94c856db4141cfb7ada7515d5b6e9e
1024 19:48:22.156261 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.156340 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2304656}
W20251024 19:48:22.156965 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9a94c856db4141cfb7ada7515d5b6e9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a94c856db4141cfb7ada7515d5b6e9e already in progress: creating tablet
I20251024 19:48:22.157054 1774 tablet_replica.cc:333] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.157258 1774 raft_consensus.cc:2243] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.157456 1774 raft_consensus.cc:2272] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.157652 1775 ts_tablet_manager.cc:1916] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.158174 1774 ts_tablet_manager.cc:1916] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.160900 1767 tablet_bootstrap.cc:492] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:22.161026 1768 raft_consensus.cc:359] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:22.163555 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 991) took 3189 ms. Trace:
I20251024 19:48:22.163705 1776 rpcz_store.cc:276] 1024 19:48:18.973990 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.974864 (+ 874us) service_pool.cc:225] Handling call
1024 19:48:21.973283 (+2998419us) ts_tablet_manager.cc:1151] Deleting tablet cb0405765c6146cbbb187af96a60be2e
1024 19:48:21.973301 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.014442 (+ 41141us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.063340 (+ 48898us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.163534 (+100194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2998179}
I20251024 19:48:22.164153 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet cb0405765c6146cbbb187af96a60be2e (table default.Table_3f7f7b5dfa674b19ad19c339c619326c [id=7faf1f110af34aa8939ad42e64f60011]) successfully deleted
I20251024 19:48:22.164211 1767 tablet_bootstrap.cc:654] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.164597 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 992) took 3190 ms. Trace:
I20251024 19:48:22.164741 1776 rpcz_store.cc:276] 1024 19:48:18.974316 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.977279 (+ 2963us) service_pool.cc:225] Handling call
1024 19:48:22.164482 (+3187203us) ts_tablet_manager.cc:1151] Deleting tablet 346cb0bb8a0d43f7a15b788a303a6bcf
1024 19:48:22.164493 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.164583 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3186932}
W20251024 19:48:22.165202 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 346cb0bb8a0d43f7a15b788a303a6bcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 346cb0bb8a0d43f7a15b788a303a6bcf already in progress: creating tablet
I20251024 19:48:22.165585 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 993) took 3191 ms. Trace:
I20251024 19:48:22.165905 1775 ts_tablet_manager.cc:1929] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.165908 1776 rpcz_store.cc:276] 1024 19:48:18.974412 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:18.975117 (+ 705us) service_pool.cc:225] Handling call
1024 19:48:22.165475 (+3190358us) ts_tablet_manager.cc:1151] Deleting tablet af918843c74940888c709d9a05aa6e0f
1024 19:48:22.165483 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.165568 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3169595}
I20251024 19:48:22.166095 1775 log.cc:1199] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/d840888183ec4ef7bb245363c5bac322
W20251024 19:48:22.166412 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet af918843c74940888c709d9a05aa6e0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af918843c74940888c709d9a05aa6e0f already in progress: creating tablet
I20251024 19:48:22.166780 1776 tablet_replica.cc:333] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:22.166998 1776 raft_consensus.cc:2243] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:22.167385 1776 raft_consensus.cc:2272] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.161528 1768 raft_consensus.cc:385] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.167906 1768 raft_consensus.cc:740] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.168226 1768 consensus_queue.cc:260] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:22.168876 1768 ts_tablet_manager.cc:1434] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251024 19:48:22.169392 1775 ts_tablet_manager.cc:1950] T d840888183ec4ef7bb245363c5bac322 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:22.170279 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1006) took 2397 ms. Trace:
I20251024 19:48:22.170823 1775 rpcz_store.cc:276] 1024 19:48:19.772393 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.772451 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:22.155225 (+2382774us) ts_tablet_manager.cc:1151] Deleting tablet d840888183ec4ef7bb245363c5bac322
1024 19:48:22.155234 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.160857 (+ 5623us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.165756 (+ 4899us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.170236 (+ 4480us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2377856}
I20251024 19:48:22.171300 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet d840888183ec4ef7bb245363c5bac322 (table default.Table_cb99a44a90e849be948f4a481c07a91d [id=5edc315e5e43438f83c57c449df681a8]) successfully deleted
I20251024 19:48:22.171665 1775 tablet_replica.cc:333] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:22.172049 1775 raft_consensus.cc:2243] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.172305 1775 raft_consensus.cc:2272] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.173143 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6f21add4230f4d1692ce71616d05ff7a" } new_table_name: "default.Table_d92ce2f682b547aca4675850f59c89bc" modify_external_catalogs: true
I20251024 19:48:22.175400 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ce374bf694604250b3bbef15856ad01d
I20251024 19:48:22.178838 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet ce374bf694604250b3bbef15856ad01d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:22.181794 1775 ts_tablet_manager.cc:1916] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.182209 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_358a359f16ae4eeeb2972452b92e532d" } new_table_name: "default.Table_40bc5f8e5a2d4aeeb9ce677ed9dd0ccb" modify_external_catalogs: true
I20251024 19:48:22.183300 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet ce374bf694604250b3bbef15856ad01d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:22.183516 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet ce374bf694604250b3bbef15856ad01d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.189015 1775 ts_tablet_manager.cc:1929] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:22.189203 1775 log.cc:1199] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/59956dad06064024ab171631ef6c1dca
I20251024 19:48:22.189630 1775 ts_tablet_manager.cc:1950] T 59956dad06064024ab171631ef6c1dca P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:22.199231 3137 consensus_queue.cc:1048] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:22.201511 1586 consensus_peers.cc:597] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6424b2347ef642feb38a9596f2d2be1d. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:22.202940 1586 consensus_peers.cc:597] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:22.208828 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1007) took 2427 ms. Trace:
I20251024 19:48:22.209470 1775 rpcz_store.cc:276] 1024 19:48:19.772870 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.772921 (+ 51us) service_pool.cc:225] Handling call
1024 19:48:22.171599 (+2398678us) ts_tablet_manager.cc:1151] Deleting tablet 59956dad06064024ab171631ef6c1dca
1024 19:48:22.171609 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.185635 (+ 14026us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.188892 (+ 3257us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.200671 (+ 11779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2394168}
I20251024 19:48:22.210510 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1008) took 2381 ms. Trace:
I20251024 19:48:22.210526 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 59956dad06064024ab171631ef6c1dca (table default.Table_9c03ce6b892d44fcbff5c92f613bc37f [id=e51bd6230e2e41429af7e8b08198bbe1]) successfully deleted
I20251024 19:48:22.211201 1775 rpcz_store.cc:276] 1024 19:48:19.829069 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.829129 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:22.210290 (+2381161us) ts_tablet_manager.cc:1151] Deleting tablet 9a94c856db4141cfb7ada7515d5b6e9e
1024 19:48:22.210301 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.210406 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2380911}
W20251024 19:48:22.212503 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9a94c856db4141cfb7ada7515d5b6e9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a94c856db4141cfb7ada7515d5b6e9e already in progress: creating tablet
I20251024 19:48:22.214461 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1009) took 2239 ms. Trace:
I20251024 19:48:22.215463 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet d9367428428648f7ae07722ea09d70a7
I20251024 19:48:22.215377 1775 rpcz_store.cc:276] 1024 19:48:19.975067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.975120 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:22.214250 (+2239130us) ts_tablet_manager.cc:1151] Deleting tablet f6df45dfab124358b8669bb4f01a6dd5
1024 19:48:22.214261 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.214372 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2238908}
W20251024 19:48:22.216060 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet f6df45dfab124358b8669bb4f01a6dd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6df45dfab124358b8669bb4f01a6dd5 already in progress: creating tablet
I20251024 19:48:22.216347 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet d9367428428648f7ae07722ea09d70a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:22.217188 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet d9367428428648f7ae07722ea09d70a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.217231 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet d9367428428648f7ae07722ea09d70a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:22.222646 1766 tablet_bootstrap.cc:492] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:22.222647 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a1031d785b646b095ece9be666d2a77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.222553 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_260350863f8e488f85b754a90b8facdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:22.223129 1766 ts_tablet_manager.cc:1403] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.094s user 0.004s sys 0.000s
W20251024 19:48:22.223166 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_260350863f8e488f85b754a90b8facdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.223165 1775 tablet_replica.cc:333] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:22.223829 1775 raft_consensus.cc:2243] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.228298 1775 raft_consensus.cc:2272] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.229846 1768 tablet_bootstrap.cc:492] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:22.231739 1775 ts_tablet_manager.cc:1916] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.232568 1768 tablet_bootstrap.cc:654] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:22.233709 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 64337.2us
I20251024 19:48:22.234509 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_48d22723ce864c7bbd947ce17e8e02f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.235186 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48d22723ce864c7bbd947ce17e8e02f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.238426 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_155f8d10b0d84715a1308d35a0d2b33a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.238966 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_155f8d10b0d84715a1308d35a0d2b33a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.251847 1767 tablet_bootstrap.cc:492] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:22.252290 1767 ts_tablet_manager.cc:1403] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.091s user 0.003s sys 0.001s
I20251024 19:48:22.254103 1767 raft_consensus.cc:359] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.255019 1767 raft_consensus.cc:385] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.255231 1767 raft_consensus.cc:740] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.255623 1767 consensus_queue.cc:260] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.256752 1767 ts_tablet_manager.cc:1434] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251024 19:48:22.245904 1766 raft_consensus.cc:359] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:22.255391 2858 raft_consensus.cc:493] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.263375 2858 raft_consensus.cc:515] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:22.264143 2858 leader_election.cc:290] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:22.261515 1776 ts_tablet_manager.cc:1916] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.264840 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccd46da86fe444baa675d3391cdd410c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:22.267208 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccd46da86fe444baa675d3391cdd410c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:22.267871 1444 leader_election.cc:343] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:22.268505 1447 leader_election.cc:343] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:22.268715 1447 leader_election.cc:304] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:22.269757 1767 tablet_bootstrap.cc:492] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:22.270340 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_93d0f441b9cb47e9a13b6cb7f04c4ac9" } new_table_name: "default.Table_173dabaad63847f4a9b92bcbf2bc0429" modify_external_catalogs: true
I20251024 19:48:22.271315 1767 tablet_bootstrap.cc:654] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:22.273557 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96e8588e092f45c2b10f4e1e97408c47 consensus state has no leader
I20251024 19:48:22.273964 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_cd1a44e3831a434fa5c98641671fba48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.274510 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd1a44e3831a434fa5c98641671fba48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.278208 2858 raft_consensus.cc:2749] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:22.280784 1776 ts_tablet_manager.cc:1929] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.280906 1776 log.cc:1199] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/3cf4596b333244cc9e9e35a128721e4a
I20251024 19:48:22.281316 1776 ts_tablet_manager.cc:1950] T 3cf4596b333244cc9e9e35a128721e4a P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:22.281996 3137 consensus_queue.cc:1048] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:22.282588 1774 ts_tablet_manager.cc:1929] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.282704 1774 log.cc:1199] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/62667092d7e244cca1020ec220ea9124
I20251024 19:48:22.283058 1774 ts_tablet_manager.cc:1950] T 62667092d7e244cca1020ec220ea9124 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:22.283874 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 939) took 2364 ms. Trace:
I20251024 19:48:22.283967 1774 rpcz_store.cc:276] 1024 19:48:19.918956 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.919008 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:22.156998 (+2237990us) ts_tablet_manager.cc:1151] Deleting tablet 62667092d7e244cca1020ec220ea9124
1024 19:48:22.157007 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.273982 (+116975us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.282474 (+ 8492us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.283852 (+ 1378us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2237726}
I20251024 19:48:22.284390 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 62667092d7e244cca1020ec220ea9124 (table default.Table_06b07944861041f8a66001a6555d7659 [id=96c4981300dc48e6bf55fc633164ed63]) successfully deleted
I20251024 19:48:22.292881 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_596ac12622974095b23e65278e8ba38b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.293275 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bec3806757214c869cf1c4cecd7cc26e has no consensus state
W20251024 19:48:22.293602 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_596ac12622974095b23e65278e8ba38b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.293637 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0dd268e924c14ae1995b078b85afca08" } new_table_name: "default.Table_05d774ae8eb94b3790d1e74b633c8415" modify_external_catalogs: true
I20251024 19:48:22.295164 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7b5a349ef82a44d0861177dc73afad8d" } new_table_name: "default.Table_1de00b2467ce430ca8a359da338d327e" modify_external_catalogs: true
I20251024 19:48:22.296978 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7cacdff7332349bca13cbc786bb5b346"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.297565 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cacdff7332349bca13cbc786bb5b346 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.299254 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1016) took 2939 ms. Trace:
W20251024 19:48:22.299794 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9ee0844646e4034b45f307738623968 has no consensus state
I20251024 19:48:22.302268 1776 rpcz_store.cc:276] 1024 19:48:19.359352 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.359747 (+ 395us) service_pool.cc:225] Handling call
1024 19:48:22.166736 (+2806989us) ts_tablet_manager.cc:1151] Deleting tablet 3cf4596b333244cc9e9e35a128721e4a
1024 19:48:22.166746 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.268825 (+102079us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.280668 (+ 11843us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.299227 (+ 18559us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2806602}
I20251024 19:48:22.304183 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 3cf4596b333244cc9e9e35a128721e4a (table default.Table_25adc9559f2b43e2a4adc92dfc0c7807 [id=60d23d1c07704c33bc979f7ba025b102]) successfully deleted
I20251024 19:48:22.304224 1776 tablet_replica.cc:333] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:48:22.310559 1586 consensus_peers.cc:597] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 129debb55f4a4b7295a8ad88e55ebc83. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:22.311825 1776 raft_consensus.cc:2243] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:22.314539 1776 raft_consensus.cc:2272] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.318352 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet 54bc7c7764584e11948d2831a4e5e0b6 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:22.319092 1776 ts_tablet_manager.cc:1916] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.315718 1767 tablet_bootstrap.cc:492] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:22.320060 1767 ts_tablet_manager.cc:1403] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.000s
I20251024 19:48:22.321223 1767 raft_consensus.cc:359] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.289556 1775 ts_tablet_manager.cc:1929] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251024 19:48:22.324903 1775 log.cc:1199] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/e03c488d0f264684b0897a81a04d7466
I20251024 19:48:22.326531 1767 raft_consensus.cc:385] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.327029 1775 ts_tablet_manager.cc:1950] T e03c488d0f264684b0897a81a04d7466 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:22.327271 1767 raft_consensus.cc:740] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.327670 1767 consensus_queue.cc:260] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.319535 1766 raft_consensus.cc:385] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.330651 1767 ts_tablet_manager.cc:1434] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
W20251024 19:48:22.332602 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3cdd85a3ca7545eea953e149a6c0d572 consensus state has no leader
W20251024 19:48:22.333087 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c9ff67824254c04bd7d024cbcc6c080 consensus state has no leader
I20251024 19:48:22.333933 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_68e567e80b54411a9ff4a53e1e8961a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:22.335311 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ec9fec2116ab4e40bff85f19b11d6f73" } modify_external_catalogs: true
I20251024 19:48:22.335583 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ec9fec2116ab4e40bff85f19b11d6f73" } modify_external_catalogs: true
I20251024 19:48:22.342028 1767 tablet_bootstrap.cc:492] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:22.329699 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1010) took 2353 ms. Trace:
I20251024 19:48:22.349056 1767 tablet_bootstrap.cc:654] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.329774 1766 raft_consensus.cc:740] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.345957 1775 rpcz_store.cc:276] 1024 19:48:19.975703 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.977871 (+ 2168us) service_pool.cc:225] Handling call
1024 19:48:22.223023 (+2245152us) ts_tablet_manager.cc:1151] Deleting tablet e03c488d0f264684b0897a81a04d7466
1024 19:48:22.223035 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.286096 (+ 63061us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.289406 (+ 3310us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.329623 (+ 40217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2244977}
I20251024 19:48:22.350525 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet e03c488d0f264684b0897a81a04d7466 (table default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8]) successfully deleted
I20251024 19:48:22.351295 1775 tablet_replica.cc:333] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:22.351840 1775 raft_consensus.cc:2243] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.352385 1775 raft_consensus.cc:2272] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.353314 1775 ts_tablet_manager.cc:1916] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.350106 1766 consensus_queue.cc:260] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:22.355966 1766 ts_tablet_manager.cc:1434] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.133s user 0.000s sys 0.003s
I20251024 19:48:22.356460 1766 tablet_bootstrap.cc:492] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:22.369805 2858 raft_consensus.cc:493] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.370160 2858 raft_consensus.cc:515] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:22.371054 2858 leader_election.cc:290] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:22.348794 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 940) took 2363 ms. Trace:
I20251024 19:48:22.374248 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "780442b310e6409498e6bbe4256280a4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:22.375034 1447 leader_election.cc:343] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:22.373761 1774 rpcz_store.cc:276] 1024 19:48:19.985206 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.985295 (+ 89us) service_pool.cc:225] Handling call
1024 19:48:22.348499 (+2363204us) ts_tablet_manager.cc:1151] Deleting tablet f6df45dfab124358b8669bb4f01a6dd5
1024 19:48:22.348516 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.348760 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2363004}
I20251024 19:48:22.376818 1774 tablet_replica.cc:333] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.377058 1774 raft_consensus.cc:2243] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.364094 1768 tablet_bootstrap.cc:492] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:22.377367 1768 ts_tablet_manager.cc:1403] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.148s user 0.003s sys 0.004s
I20251024 19:48:22.378260 1768 raft_consensus.cc:359] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:22.378507 1768 raft_consensus.cc:385] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.378593 1768 raft_consensus.cc:740] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.378894 1768 consensus_queue.cc:260] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:22.379442 1768 ts_tablet_manager.cc:1434] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:22.379858 1768 tablet_bootstrap.cc:492] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:22.383054 1768 tablet_bootstrap.cc:654] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.385351 1774 raft_consensus.cc:2272] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.386168 1774 ts_tablet_manager.cc:1916] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:22.377457 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet f6df45dfab124358b8669bb4f01a6dd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6df45dfab124358b8669bb4f01a6dd5 already in progress: creating tablet
I20251024 19:48:22.394981 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4bb5f923e4074b398f16e0f53223fbf6" } modify_external_catalogs: true
I20251024 19:48:22.395224 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4bb5f923e4074b398f16e0f53223fbf6" } modify_external_catalogs: true
W20251024 19:48:22.395905 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68e567e80b54411a9ff4a53e1e8961a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.397970 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 50cea450eda9485daba40bf875a1b90c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.397451 1774 ts_tablet_manager.cc:1929] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251024 19:48:22.402388 1774 log.cc:1199] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/e03c488d0f264684b0897a81a04d7466
I20251024 19:48:22.403146 1774 ts_tablet_manager.cc:1950] T e03c488d0f264684b0897a81a04d7466 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:22.405623 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 941) took 2402 ms. Trace:
I20251024 19:48:22.406816 1774 rpcz_store.cc:276] 1024 19:48:20.002925 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.003353 (+ 428us) service_pool.cc:225] Handling call
1024 19:48:22.376766 (+2373413us) ts_tablet_manager.cc:1151] Deleting tablet e03c488d0f264684b0897a81a04d7466
1024 19:48:22.376777 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.390590 (+ 13813us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.397315 (+ 6725us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.405599 (+ 8284us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2373185}
I20251024 19:48:22.406661 1768 tablet_bootstrap.cc:492] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:22.407359 1768 ts_tablet_manager.cc:1403] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.006s
I20251024 19:48:22.408795 1768 raft_consensus.cc:359] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.409123 1768 raft_consensus.cc:385] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.409333 1768 raft_consensus.cc:740] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.409807 1768 consensus_queue.cc:260] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.413296 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet e03c488d0f264684b0897a81a04d7466 (table default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8]) successfully deleted
I20251024 19:48:22.413808 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 944) took 2306 ms. Trace:
I20251024 19:48:22.413940 1774 rpcz_store.cc:276] 1024 19:48:20.107078 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.107145 (+ 67us) service_pool.cc:225] Handling call
1024 19:48:22.413670 (+2306525us) ts_tablet_manager.cc:1151] Deleting tablet 3ef6d244c498452ca79cd6c6d5c003df
1024 19:48:22.413681 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.413791 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2306325}
W20251024 19:48:22.414423 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3ef6d244c498452ca79cd6c6d5c003df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ef6d244c498452ca79cd6c6d5c003df already in progress: creating tablet
I20251024 19:48:22.415012 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 943) took 2308 ms. Trace:
I20251024 19:48:22.415186 1774 rpcz_store.cc:276] 1024 19:48:20.106985 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.107942 (+ 957us) service_pool.cc:225] Handling call
1024 19:48:22.414886 (+2306944us) ts_tablet_manager.cc:1151] Deleting tablet 8d1b6eaa353c4284a2b114b42437272c
1024 19:48:22.414897 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.414994 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2306766}
W20251024 19:48:22.415680 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 8d1b6eaa353c4284a2b114b42437272c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1b6eaa353c4284a2b114b42437272c already in progress: creating tablet
I20251024 19:48:22.416086 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 947) took 2308 ms. Trace:
I20251024 19:48:22.416191 1774 rpcz_store.cc:276] 1024 19:48:20.108052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.108100 (+ 48us) service_pool.cc:225] Handling call
1024 19:48:22.415981 (+2307881us) ts_tablet_manager.cc:1151] Deleting tablet 3a712f52cd114e9bb07c2717e5d728fd
1024 19:48:22.415991 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.416070 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2307733}
W20251024 19:48:22.416612 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3a712f52cd114e9bb07c2717e5d728fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a712f52cd114e9bb07c2717e5d728fd already in progress: creating tablet
I20251024 19:48:22.416967 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 946) took 2309 ms. Trace:
I20251024 19:48:22.417078 1774 rpcz_store.cc:276] 1024 19:48:20.107935 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.108311 (+ 376us) service_pool.cc:225] Handling call
1024 19:48:22.416870 (+2308559us) ts_tablet_manager.cc:1151] Deleting tablet 3c708051e9754fe8afc45154b0a4f30c
1024 19:48:22.416879 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.416952 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2308442}
W20251024 19:48:22.417798 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3c708051e9754fe8afc45154b0a4f30c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c708051e9754fe8afc45154b0a4f30c already in progress: creating tablet
I20251024 19:48:22.417829 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_72458dd9d8744a1ba775cfdcd2cc6e8e" } modify_external_catalogs: true
I20251024 19:48:22.418165 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_72458dd9d8744a1ba775cfdcd2cc6e8e" } modify_external_catalogs: true
I20251024 19:48:22.418205 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 945) took 2311 ms. Trace:
I20251024 19:48:22.418280 1774 rpcz_store.cc:276] 1024 19:48:20.107176 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.107382 (+ 206us) service_pool.cc:225] Handling call
1024 19:48:22.418078 (+2310696us) ts_tablet_manager.cc:1151] Deleting tablet 9287f340fb554904986f6c18b7356701
1024 19:48:22.418087 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.418190 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2309420}
W20251024 19:48:22.419942 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9287f340fb554904986f6c18b7356701 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9287f340fb554904986f6c18b7356701 already in progress: creating tablet
I20251024 19:48:22.420308 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 949) took 2212 ms. Trace:
I20251024 19:48:22.420387 1774 rpcz_store.cc:276] 1024 19:48:20.207545 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.207594 (+ 49us) service_pool.cc:225] Handling call
1024 19:48:22.420206 (+2212612us) ts_tablet_manager.cc:1151] Deleting tablet 6b1a3523954448de93357510e6c1ec4e
1024 19:48:22.420217 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.420293 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2212418}
W20251024 19:48:22.420766 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 6b1a3523954448de93357510e6c1ec4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b1a3523954448de93357510e6c1ec4e already in progress: creating tablet
I20251024 19:48:22.421067 1774 tablet_replica.cc:333] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.420722 1766 tablet_bootstrap.cc:654] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.421329 1774 raft_consensus.cc:2243] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:22.362895 1586 consensus_peers.cc:597] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e03c488d0f264684b0897a81a04d7466. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:22.374541 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "780442b310e6409498e6bbe4256280a4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:22.425904 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 63084.8us
I20251024 19:48:22.405668 2923 raft_consensus.cc:493] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.426561 2923 raft_consensus.cc:515] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.427184 2923 leader_election.cc:290] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:22.430042 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2a68bb1dc6c483388784b9bb19ce065" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:22.430532 1366 raft_consensus.cc:3060] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:22.436000 1774 raft_consensus.cc:2272] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:22.436403 1586 consensus_peers.cc:597] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59956dad06064024ab171631ef6c1dca. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:22.436548 1774 ts_tablet_manager.cc:1916] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.437685 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_64e7af7019f24a8eaf186a4e93208f03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.438246 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64e7af7019f24a8eaf186a4e93208f03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.449928 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3ddc38c30d594d45bc5e37c0994e4dde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.450676 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ddc38c30d594d45bc5e37c0994e4dde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.451828 3137 raft_consensus.cc:493] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.451957 3137 raft_consensus.cc:515] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:22.452551 3137 leader_election.cc:290] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:22.453323 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "385e71e672154e399f14321eac0d8625" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:22.454064 1586 leader_election.cc:343] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:22.454084 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "385e71e672154e399f14321eac0d8625" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:22.458626 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_99a4ef9d7f6e40d1b12c6d8420ae7e76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.459224 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99a4ef9d7f6e40d1b12c6d8420ae7e76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.460045 1774 ts_tablet_manager.cc:1929] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:22.460143 1774 log.cc:1199] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/81a3de69a13c459b9b7c02761ab7261e
I20251024 19:48:22.460395 1774 ts_tablet_manager.cc:1950] T 81a3de69a13c459b9b7c02761ab7261e P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:48:22.465821 1444 leader_election.cc:343] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:22.466090 1444 leader_election.cc:304] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:22.466673 2858 raft_consensus.cc:2749] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:22.434553 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2a68bb1dc6c483388784b9bb19ce065" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:22.467542 1586 leader_election.cc:343] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: Tablet not RUNNING: STOPPED
I20251024 19:48:22.468659 1504 raft_consensus.cc:2468] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:22.469126 1586 leader_election.cc:304] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:22.469293 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 948) took 2261 ms. Trace:
I20251024 19:48:22.469375 1774 rpcz_store.cc:276] 1024 19:48:20.207404 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.209717 (+ 2313us) service_pool.cc:225] Handling call
1024 19:48:22.421026 (+2211309us) ts_tablet_manager.cc:1151] Deleting tablet 81a3de69a13c459b9b7c02761ab7261e
1024 19:48:22.421034 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.451459 (+ 30425us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.459954 (+ 8495us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.469273 (+ 9319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2211112}
I20251024 19:48:22.469560 3137 raft_consensus.cc:2804] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:22.469702 3137 raft_consensus.cc:493] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:22.469777 3137 raft_consensus.cc:3060] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:22.469763 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 81a3de69a13c459b9b7c02761ab7261e (table default.Table_b5a66c4e9cd8408782690b66cdb87473 [id=3d82ad2f9e184043aed4ec0b05df724f]) successfully deleted
I20251024 19:48:22.471511 1768 ts_tablet_manager.cc:1434] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.064s user 0.018s sys 0.002s
I20251024 19:48:22.472187 1776 ts_tablet_manager.cc:1929] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.472272 1776 log.cc:1199] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/129debb55f4a4b7295a8ad88e55ebc83
I20251024 19:48:22.472606 3141 raft_consensus.cc:493] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.472740 3141 raft_consensus.cc:515] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.473376 3141 leader_election.cc:290] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:22.474385 1364 raft_consensus.cc:2468] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:22.472626 1776 ts_tablet_manager.cc:1950] T 129debb55f4a4b7295a8ad88e55ebc83 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:22.475332 1586 leader_election.cc:304] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
I20251024 19:48:22.475503 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1017) took 2937 ms. Trace:
I20251024 19:48:22.475575 1776 rpcz_store.cc:276] 1024 19:48:19.537894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.537966 (+ 72us) service_pool.cc:225] Handling call
1024 19:48:22.304165 (+2766199us) ts_tablet_manager.cc:1151] Deleting tablet 129debb55f4a4b7295a8ad88e55ebc83
1024 19:48:22.304176 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.422977 (+118801us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.472075 (+ 49098us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.475485 (+ 3410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2766022}
I20251024 19:48:22.475950 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 129debb55f4a4b7295a8ad88e55ebc83 (table default.Table_c777799df90e491992909e8a45205a06 [id=d11ca7b446a44dd28c9c243a6d7787e6]) successfully deleted
I20251024 19:48:22.476373 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10891ff5bf0d4b93bcab9faa5f7346f1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:22.476647 1505 raft_consensus.cc:2468] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:22.477082 3141 raft_consensus.cc:2804] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:22.477157 3141 raft_consensus.cc:493] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:22.477216 3141 raft_consensus.cc:3060] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:22.477325 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1018) took 2846 ms. Trace:
I20251024 19:48:22.477293 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6233280a74da4d1981019506fd1be2f3" } new_table_name: "default.Table_1d47f7564be440f2b1324af2590302d8" modify_external_catalogs: true
I20251024 19:48:22.477397 1776 rpcz_store.cc:276] 1024 19:48:19.630788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.630870 (+ 82us) service_pool.cc:225] Handling call
1024 19:48:22.477205 (+2846335us) ts_tablet_manager.cc:1151] Deleting tablet e38ef2ac19d14f9c91fa8abe86cc7987
1024 19:48:22.477215 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.477309 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2846106}
I20251024 19:48:22.478236 1776 tablet_replica.cc:333] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:22.478420 1776 raft_consensus.cc:2243] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:22.479894 3141 raft_consensus.cc:515] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.480479 3141 leader_election.cc:290] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:22.481076 1767 tablet_bootstrap.cc:492] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:22.481245 1767 ts_tablet_manager.cc:1403] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.139s user 0.004s sys 0.001s
I20251024 19:48:22.481802 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10891ff5bf0d4b93bcab9faa5f7346f1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:22.482168 1767 raft_consensus.cc:359] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.482379 1767 raft_consensus.cc:385] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.482436 1767 raft_consensus.cc:740] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.482656 1767 consensus_queue.cc:260] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:22.483100 1586 leader_election.cc:343] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:22.483276 1767 ts_tablet_manager.cc:1434] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:22.483309 1586 leader_election.cc:304] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:22.483564 1767 tablet_bootstrap.cc:492] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:22.484836 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2a68bb1dc6c483388784b9bb19ce065" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:22.485395 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2a68bb1dc6c483388784b9bb19ce065" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:22.485499 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 366c17dbe45a4f8f9830dca68f621121 consensus state has no leader
I20251024 19:48:22.485549 1364 raft_consensus.cc:3060] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:22.489717 1364 raft_consensus.cc:2468] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
W20251024 19:48:22.490154 1586 leader_election.cc:343] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Illegal state: Tablet not RUNNING: STOPPED
I20251024 19:48:22.490433 1586 leader_election.cc:304] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
I20251024 19:48:22.472749 3137 raft_consensus.cc:515] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:22.491430 3137 leader_election.cc:290] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:22.491578 1776 raft_consensus.cc:2272] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.491633 3137 raft_consensus.cc:2804] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:22.491704 3137 raft_consensus.cc:493] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:22.491763 3137 raft_consensus.cc:3060] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:22.492267 1776 ts_tablet_manager.cc:1916] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.495005 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_173dabaad63847f4a9b92bcbf2bc0429" } new_table_name: "default.Table_e845d12029a948f7b361c33a3cdabfc2" modify_external_catalogs: true
I20251024 19:48:22.496196 1768 tablet_bootstrap.cc:492] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:22.496814 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c93f1a18914a43839b3bf865dd238a1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.497480 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c93f1a18914a43839b3bf865dd238a1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.498267 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9fcc7f42f92641fdbef94211601c4a32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:22.498772 1774 tablet_replica.cc:333] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
W20251024 19:48:22.498800 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fcc7f42f92641fdbef94211601c4a32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.498965 1774 raft_consensus.cc:2243] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.499125 1774 raft_consensus.cc:2272] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.499702 1774 ts_tablet_manager.cc:1916] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.500356 1768 tablet_bootstrap.cc:654] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.500363 3141 raft_consensus.cc:2804] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:22.501205 3141 raft_consensus.cc:697] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:22.501523 3141 consensus_queue.cc:237] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.504503 1366 raft_consensus.cc:3060] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:22.504858 1767 tablet_bootstrap.cc:654] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.507344 1774 ts_tablet_manager.cc:1929] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.507513 1774 log.cc:1199] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/5d129bdc876e425aabcf1c99997e319a
I20251024 19:48:22.507898 1774 ts_tablet_manager.cc:1950] T 5d129bdc876e425aabcf1c99997e319a P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:22.508723 3137 raft_consensus.cc:515] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.508777 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 950) took 2268 ms. Trace:
W20251024 19:48:22.509032 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9ee0844646e4034b45f307738623968 has no consensus state
I20251024 19:48:22.508867 1774 rpcz_store.cc:276] 1024 19:48:20.240112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.240177 (+ 65us) service_pool.cc:225] Handling call
1024 19:48:22.472739 (+2232562us) ts_tablet_manager.cc:1151] Deleting tablet 5d129bdc876e425aabcf1c99997e319a
1024 19:48:22.498718 (+ 25979us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.503883 (+ 5165us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.507247 (+ 3364us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.508754 (+ 1507us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2232349}
I20251024 19:48:22.509521 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 5d129bdc876e425aabcf1c99997e319a (table default.Table_a4b63032d2224a6d95765cb93137f4a2 [id=d28fd80220c64f8496a09f5d4e712874]) successfully deleted
I20251024 19:48:22.510553 3137 leader_election.cc:290] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:22.511363 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_160ec7727d65458cbbf48a790008ba6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.511861 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_160ec7727d65458cbbf48a790008ba6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.513463 1774 tablet_replica.cc:333] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.515040 1768 tablet_bootstrap.cc:492] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:22.515190 1768 ts_tablet_manager.cc:1403] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.002s
I20251024 19:48:22.516374 1768 raft_consensus.cc:359] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.516530 1768 raft_consensus.cc:385] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.516572 1768 raft_consensus.cc:740] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.516731 1768 consensus_queue.cc:260] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.517108 1768 ts_tablet_manager.cc:1434] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251024 19:48:22.517328 1768 tablet_bootstrap.cc:492] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:22.517800 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet d2a68bb1dc6c483388784b9bb19ce065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.513720 1774 raft_consensus.cc:2243] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.518127 1774 raft_consensus.cc:2272] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.518627 1774 ts_tablet_manager.cc:1916] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.518576 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_391d10d0c6f44940bf592166d0b1aa40" } new_table_name: "default.Table_5246796e5d1f4fc9b7e766d6bfa2ae29" modify_external_catalogs: true
I20251024 19:48:22.518886 1776 ts_tablet_manager.cc:1929] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.518983 1776 log.cc:1199] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/6424b2347ef642feb38a9596f2d2be1d
I20251024 19:48:22.519312 1776 ts_tablet_manager.cc:1950] T 6424b2347ef642feb38a9596f2d2be1d P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:22.525547 1768 tablet_bootstrap.cc:654] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.527413 1767 tablet_bootstrap.cc:492] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:22.527706 1767 ts_tablet_manager.cc:1403] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.007s
I20251024 19:48:22.528851 1767 raft_consensus.cc:359] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.529067 1767 raft_consensus.cc:385] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.529151 1767 raft_consensus.cc:740] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.529364 1767 consensus_queue.cc:260] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.529980 1767 ts_tablet_manager.cc:1434] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20251024 19:48:22.540702 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e111980a15894b5084d9384297389b5d has no consensus state
I20251024 19:48:22.545433 1768 tablet_bootstrap.cc:492] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:22.545663 1768 ts_tablet_manager.cc:1403] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.004s
I20251024 19:48:22.546022 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet d840888183ec4ef7bb245363c5bac322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.546572 1768 raft_consensus.cc:359] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.546794 1768 raft_consensus.cc:385] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.546854 1768 raft_consensus.cc:740] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.547075 1768 consensus_queue.cc:260] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.547693 1768 ts_tablet_manager.cc:1434] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:22.551663 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1019) took 2883 ms. Trace:
I20251024 19:48:22.551767 1776 rpcz_store.cc:276] 1024 19:48:19.668111 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.668178 (+ 67us) service_pool.cc:225] Handling call
1024 19:48:22.478194 (+2810016us) ts_tablet_manager.cc:1151] Deleting tablet 6424b2347ef642feb38a9596f2d2be1d
1024 19:48:22.478203 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.515723 (+ 37520us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.518803 (+ 3080us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.551623 (+ 32820us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2809821}
I20251024 19:48:22.552090 1776 tablet_replica.cc:333] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:22.552278 1776 raft_consensus.cc:2243] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:22.552628 1776 raft_consensus.cc:2272] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.553311 1776 ts_tablet_manager.cc:1916] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.553869 1767 tablet_bootstrap.cc:492] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:22.554335 1774 ts_tablet_manager.cc:1929] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:22.554472 1774 log.cc:1199] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/b20a6845ae1645c0b51f096d471d4ccc
I20251024 19:48:22.554823 1774 ts_tablet_manager.cc:1950] T b20a6845ae1645c0b51f096d471d4ccc P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:22.555565 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 951) took 2151 ms. Trace:
I20251024 19:48:22.555660 1774 rpcz_store.cc:276] 1024 19:48:20.404262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.404318 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:22.513076 (+2108758us) ts_tablet_manager.cc:1151] Deleting tablet b20a6845ae1645c0b51f096d471d4ccc
1024 19:48:22.513412 (+ 336us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.538224 (+ 24812us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.554234 (+ 16010us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.555546 (+ 1312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2108571}
I20251024 19:48:22.556042 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 952) took 2133 ms. Trace:
I20251024 19:48:22.556121 1774 rpcz_store.cc:276] 1024 19:48:20.422678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.422761 (+ 83us) service_pool.cc:225] Handling call
1024 19:48:22.555938 (+2133177us) ts_tablet_manager.cc:1151] Deleting tablet fc187c8f34594f4fb37d8afe0855263b
1024 19:48:22.555946 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.556029 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2132954}
I20251024 19:48:22.556185 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet b20a6845ae1645c0b51f096d471d4ccc (table default.Table_50e12cf5f39d4fa2983835ef0dbbc3d6 [id=201e5232763a4940a1288a56a695f71c]) successfully deleted
W20251024 19:48:22.556715 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet fc187c8f34594f4fb37d8afe0855263b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc187c8f34594f4fb37d8afe0855263b already in progress: creating tablet
I20251024 19:48:22.556771 1767 tablet_bootstrap.cc:654] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.558214 1774 tablet_replica.cc:333] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.558444 1774 raft_consensus.cc:2243] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.558602 1774 raft_consensus.cc:2272] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.559140 1774 ts_tablet_manager.cc:1916] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.560036 1768 tablet_bootstrap.cc:492] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:22.562661 1768 tablet_bootstrap.cc:654] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.566099 1774 ts_tablet_manager.cc:1929] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.566231 1774 log.cc:1199] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/844638a6b6944d0394df3e78a8ea51d6
I20251024 19:48:22.566679 1774 ts_tablet_manager.cc:1950] T 844638a6b6944d0394df3e78a8ea51d6 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:22.567612 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 953) took 2097 ms. Trace:
I20251024 19:48:22.567716 1774 rpcz_store.cc:276] 1024 19:48:20.470316 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.470382 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:22.558163 (+2087781us) ts_tablet_manager.cc:1151] Deleting tablet 844638a6b6944d0394df3e78a8ea51d6
1024 19:48:22.558179 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.562412 (+ 4233us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.565979 (+ 3567us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.567587 (+ 1608us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2034271}
I20251024 19:48:22.568076 1774 tablet_replica.cc:333] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.568260 1774 raft_consensus.cc:2243] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.568521 1774 raft_consensus.cc:2272] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.569135 1774 ts_tablet_manager.cc:1916] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.569636 1768 tablet_bootstrap.cc:492] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:22.569756 1768 ts_tablet_manager.cc:1403] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20251024 19:48:22.570036 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 844638a6b6944d0394df3e78a8ea51d6 (table default.Table_823526f1c3164a3ba70c44dd8388fb9c [id=fc5fe814531e4110ace3fb312aefa7ac]) successfully deleted
I20251024 19:48:22.570468 1776 ts_tablet_manager.cc:1929] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.570385 1768 raft_consensus.cc:359] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.570557 1768 raft_consensus.cc:385] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.570585 1776 log.cc:1199] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/50cea450eda9485daba40bf875a1b90c
I20251024 19:48:22.570598 1768 raft_consensus.cc:740] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.570766 1768 consensus_queue.cc:260] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.572822 1776 ts_tablet_manager.cc:1950] T 50cea450eda9485daba40bf875a1b90c P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:22.575256 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1020) took 2766 ms. Trace:
I20251024 19:48:22.575349 1776 rpcz_store.cc:276] 1024 19:48:19.808834 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.808890 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:22.552039 (+2743149us) ts_tablet_manager.cc:1151] Deleting tablet 50cea450eda9485daba40bf875a1b90c
1024 19:48:22.552050 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.561334 (+ 9284us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.570341 (+ 9007us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.575235 (+ 4894us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2736760}
I20251024 19:48:22.575795 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1021) took 2754 ms. Trace:
I20251024 19:48:22.575872 1776 rpcz_store.cc:276] 1024 19:48:19.821388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.821443 (+ 55us) service_pool.cc:225] Handling call
1024 19:48:22.575640 (+2754197us) ts_tablet_manager.cc:1151] Deleting tablet 9a94c856db4141cfb7ada7515d5b6e9e
1024 19:48:22.575652 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.575775 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2753972}
I20251024 19:48:22.576186 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1022) took 2646 ms. Trace:
I20251024 19:48:22.576267 1776 rpcz_store.cc:276] 1024 19:48:19.930059 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.930119 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:22.576087 (+2645968us) ts_tablet_manager.cc:1151] Deleting tablet f6df45dfab124358b8669bb4f01a6dd5
1024 19:48:22.576097 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.576173 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2645764}
I20251024 19:48:22.576566 1776 tablet_replica.cc:333] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:22.576769 1768 ts_tablet_manager.cc:1434] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251024 19:48:22.576763 1776 raft_consensus.cc:2243] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:22.580598 1776 raft_consensus.cc:2272] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.581187 1768 tablet_bootstrap.cc:492] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:22.581195 1776 ts_tablet_manager.cc:1916] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.582022 3296 raft_consensus.cc:493] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.582139 3296 raft_consensus.cc:515] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.582672 3296 leader_election.cc:290] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:22.583873 1308 leader_election.cc:343] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: d840888183ec4ef7bb245363c5bac322
I20251024 19:48:22.590507 1768 tablet_bootstrap.cc:654] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.594161 1011 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskc9NInd/build/asan/bin/kudu pid 1087
I20251024 19:48:22.596488 1774 ts_tablet_manager.cc:1929] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:22.596606 1774 log.cc:1199] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/a3259823b622461fa823841925b3d203
I20251024 19:48:22.597023 1774 ts_tablet_manager.cc:1950] T a3259823b622461fa823841925b3d203 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:48:22.598135 1708 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.AlterTable -> {remote=127.0.252.252:36627, user_credentials={real_user=slave}} blocked reactor thread for 56382us
I20251024 19:48:22.601334 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a9b2002bb5a544c698822168e561c8cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.602053 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9b2002bb5a544c698822168e561c8cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.603813 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f3fbc16ea2444f798f1e23a68acdf61f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.604328 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3fbc16ea2444f798f1e23a68acdf61f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.604658 1768 tablet_bootstrap.cc:492] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:22.604785 1768 ts_tablet_manager.cc:1403] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.007s
I20251024 19:48:22.605635 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8839c58c3b974b41938ec5bba438af46" } new_table_name: "default.Table_5cc721059c6a427c81d0cc1b58e5b57d" modify_external_catalogs: true
I20251024 19:48:22.615533 1776 ts_tablet_manager.cc:1929] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251024 19:48:22.616384 1776 log.cc:1199] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/e03c488d0f264684b0897a81a04d7466
I20251024 19:48:22.616307 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_09c35871ae6345f8bd692a8e416323e5" } new_table_name: "default.Table_9740a4661e9345ccbf754fbd167df7f6" modify_external_catalogs: true
I20251024 19:48:22.617158 1776 ts_tablet_manager.cc:1950] T e03c488d0f264684b0897a81a04d7466 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:22.617108 2858 raft_consensus.cc:493] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.617707 2858 raft_consensus.cc:515] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:22.619016 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4100abc9ff12471e9a4959b6b24f7e99" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:22.619439 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4100abc9ff12471e9a4959b6b24f7e99" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:22.619660 1447 leader_election.cc:343] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:22.619822 2858 leader_election.cc:290] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:22.620105 1444 leader_election.cc:343] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:22.620244 1444 leader_election.cc:304] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:22.620749 2858 raft_consensus.cc:2749] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:22.622872 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4f22d5090caa447d9715e95a8a248432"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.623675 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f22d5090caa447d9715e95a8a248432 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.624723 1768 raft_consensus.cc:359] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.624835 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 954) took 2134 ms. Trace:
I20251024 19:48:22.624908 1768 raft_consensus.cc:385] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.625020 1768 raft_consensus.cc:740] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.625227 1768 consensus_queue.cc:260] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.625761 1768 ts_tablet_manager.cc:1434] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20251024 19:48:22.626044 1768 tablet_bootstrap.cc:492] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:22.627615 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3405f63ee04404298b1f0e0527522e3 consensus state has no leader
I20251024 19:48:22.628065 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d1092b6bac8f4c90a5eb12dd16060570"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.628633 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1092b6bac8f4c90a5eb12dd16060570 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.628764 1768 tablet_bootstrap.cc:654] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.624905 1774 rpcz_store.cc:276] 1024 19:48:20.490298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.490373 (+ 75us) service_pool.cc:225] Handling call
1024 19:48:22.568028 (+2077655us) ts_tablet_manager.cc:1151] Deleting tablet a3259823b622461fa823841925b3d203
1024 19:48:22.568038 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.585749 (+ 17711us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.596385 (+ 10636us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.624819 (+ 28434us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26958,"tablet-delete.queue_time_us":2045846}
I20251024 19:48:22.630158 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 955) took 2096 ms. Trace:
I20251024 19:48:22.630221 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet a3259823b622461fa823841925b3d203 (table default.Table_ea0d3589c13a4ebeb950ddc737a9948b [id=350b37bf42194fcfbb660bffe83e47c4]) successfully deleted
I20251024 19:48:22.630234 1774 rpcz_store.cc:276] 1024 19:48:20.533957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.534018 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:22.630038 (+2096020us) ts_tablet_manager.cc:1151] Deleting tablet a2cc3f3d02324c1e867b8fc159ad6d20
1024 19:48:22.630048 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.630141 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2095798}
I20251024 19:48:22.630496 1774 tablet_replica.cc:333] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.630669 1774 raft_consensus.cc:2243] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.630805 1774 raft_consensus.cc:2272] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:22.631112 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ff0ec53ae71461baf2b8f9ec065530f consensus state has no leader
I20251024 19:48:22.631314 1774 ts_tablet_manager.cc:1916] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.632357 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1023) took 2658 ms. Trace:
W20251024 19:48:22.632872 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet a2cc3f3d02324c1e867b8fc159ad6d20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2cc3f3d02324c1e867b8fc159ad6d20 already in progress: creating tablet
I20251024 19:48:22.632896 1776 rpcz_store.cc:276] 1024 19:48:19.973461 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:19.973519 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:22.576516 (+2602997us) ts_tablet_manager.cc:1151] Deleting tablet e03c488d0f264684b0897a81a04d7466
1024 19:48:22.576526 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.593756 (+ 17230us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.615431 (+ 21675us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.632332 (+ 16901us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2602742}
I20251024 19:48:22.634035 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1024) took 2543 ms. Trace:
I20251024 19:48:22.638144 1774 ts_tablet_manager.cc:1929] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:22.638263 1774 log.cc:1199] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/94fc8c5716af47faa4e831f05fce8610
I20251024 19:48:22.638641 1774 ts_tablet_manager.cc:1950] T 94fc8c5716af47faa4e831f05fce8610 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:22.639425 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 956) took 1981 ms. Trace:
I20251024 19:48:22.639518 1774 rpcz_store.cc:276] 1024 19:48:20.658371 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.658431 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:22.630453 (+1972022us) ts_tablet_manager.cc:1151] Deleting tablet 94fc8c5716af47faa4e831f05fce8610
1024 19:48:22.630462 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.634852 (+ 4390us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.638065 (+ 3213us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.639396 (+ 1331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1971778}
I20251024 19:48:22.639914 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 94fc8c5716af47faa4e831f05fce8610 (table default.Table_023e47eac6c24e328d5a01fcf3b2f579 [id=048d560cd32a44c5abf12bde27fe35c7]) successfully deleted
I20251024 19:48:22.640166 1774 tablet_replica.cc:333] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.640327 1774 raft_consensus.cc:2243] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.640475 1774 raft_consensus.cc:2272] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.640980 1774 ts_tablet_manager.cc:1916] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.648566 1774 ts_tablet_manager.cc:1929] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.648680 1774 log.cc:1199] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/cb0405765c6146cbbb187af96a60be2e
I20251024 19:48:22.649065 1774 ts_tablet_manager.cc:1950] T cb0405765c6146cbbb187af96a60be2e P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:22.649925 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 957) took 1749 ms. Trace:
I20251024 19:48:22.650022 1774 rpcz_store.cc:276] 1024 19:48:20.900178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.900251 (+ 73us) service_pool.cc:225] Handling call
1024 19:48:22.640125 (+1739874us) ts_tablet_manager.cc:1151] Deleting tablet cb0405765c6146cbbb187af96a60be2e
1024 19:48:22.640134 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.644786 (+ 4652us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.648474 (+ 3688us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.649904 (+ 1430us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1739603}
I20251024 19:48:22.650956 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet cb0405765c6146cbbb187af96a60be2e (table default.Table_3f7f7b5dfa674b19ad19c339c619326c [id=7faf1f110af34aa8939ad42e64f60011]) successfully deleted
I20251024 19:48:22.651368 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 964) took 1515 ms. Trace:
I20251024 19:48:22.651456 1774 rpcz_store.cc:276] 1024 19:48:21.136194 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.136391 (+ 197us) service_pool.cc:225] Handling call
1024 19:48:22.651225 (+1514834us) ts_tablet_manager.cc:1151] Deleting tablet 04ae62c9bf6f44108d57d0760283d063
1024 19:48:22.651237 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.651351 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1514627}
W20251024 19:48:22.651962 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 04ae62c9bf6f44108d57d0760283d063 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04ae62c9bf6f44108d57d0760283d063 already in progress: creating tablet
I20251024 19:48:22.652423 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 965) took 1516 ms. Trace:
I20251024 19:48:22.652508 1774 rpcz_store.cc:276] 1024 19:48:21.136332 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.136635 (+ 303us) service_pool.cc:225] Handling call
1024 19:48:22.652302 (+1515667us) ts_tablet_manager.cc:1151] Deleting tablet 7ecf3276437a43cf84a47a0e66553e0a
1024 19:48:22.652312 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.652394 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1515513}
W20251024 19:48:22.652937 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7ecf3276437a43cf84a47a0e66553e0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ecf3276437a43cf84a47a0e66553e0a already in progress: creating tablet
I20251024 19:48:22.653280 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 966) took 1283 ms. Trace:
I20251024 19:48:22.653355 1774 rpcz_store.cc:276] 1024 19:48:21.369855 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.369925 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:22.653193 (+1283268us) ts_tablet_manager.cc:1151] Deleting tablet 30bce965315d43948ecb8282c829c557
1024 19:48:22.653201 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.653266 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1222137}
W20251024 19:48:22.653723 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 30bce965315d43948ecb8282c829c557 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30bce965315d43948ecb8282c829c557 already in progress: creating tablet
I20251024 19:48:22.654034 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 967) took 1266 ms. Trace:
I20251024 19:48:22.654103 1774 rpcz_store.cc:276] 1024 19:48:21.387701 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.387768 (+ 67us) service_pool.cc:225] Handling call
1024 19:48:22.653945 (+1266177us) ts_tablet_manager.cc:1151] Deleting tablet 672b571081ea4cd78028b6a25625103c
1024 19:48:22.653953 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.654020 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1222815}
W20251024 19:48:22.654495 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 672b571081ea4cd78028b6a25625103c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 672b571081ea4cd78028b6a25625103c already in progress: creating tablet
I20251024 19:48:22.654834 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 968) took 1265 ms. Trace:
I20251024 19:48:22.654908 1774 rpcz_store.cc:276] 1024 19:48:21.389405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.389726 (+ 321us) service_pool.cc:225] Handling call
1024 19:48:22.654724 (+1264998us) ts_tablet_manager.cc:1151] Deleting tablet e0b57403cdf947b390a44a9889f2a2c3
1024 19:48:22.654733 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.654819 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1223496}
W20251024 19:48:22.655256 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet e0b57403cdf947b390a44a9889f2a2c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0b57403cdf947b390a44a9889f2a2c3 already in progress: creating tablet
I20251024 19:48:22.655577 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 969) took 1260 ms. Trace:
I20251024 19:48:22.655654 1774 rpcz_store.cc:276] 1024 19:48:21.394961 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.395024 (+ 63us) service_pool.cc:225] Handling call
1024 19:48:22.655488 (+1260464us) ts_tablet_manager.cc:1151] Deleting tablet 3a99e9b3fe044c7eb14fa9bffa903873
1024 19:48:22.655497 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.655563 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224206}
W20251024 19:48:22.656015 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3a99e9b3fe044c7eb14fa9bffa903873 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a99e9b3fe044c7eb14fa9bffa903873 already in progress: creating tablet
I20251024 19:48:22.656320 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 970) took 1261 ms. Trace:
I20251024 19:48:22.656554 1774 rpcz_store.cc:276] 1024 19:48:21.395077 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.395266 (+ 189us) service_pool.cc:225] Handling call
1024 19:48:22.656232 (+1260966us) ts_tablet_manager.cc:1151] Deleting tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253
1024 19:48:22.656241 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.656306 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224844}
W20251024 19:48:22.656929 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 already in progress: creating tablet
I20251024 19:48:22.657266 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 971) took 1262 ms. Trace:
I20251024 19:48:22.657335 1774 rpcz_store.cc:276] 1024 19:48:21.395196 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.395495 (+ 299us) service_pool.cc:225] Handling call
1024 19:48:22.657187 (+1261692us) ts_tablet_manager.cc:1151] Deleting tablet 59eca6bb1dd84c67a7e53131abaa937a
1024 19:48:22.657195 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.657253 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1225736}
W20251024 19:48:22.657845 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 59eca6bb1dd84c67a7e53131abaa937a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59eca6bb1dd84c67a7e53131abaa937a already in progress: creating tablet
I20251024 19:48:22.658156 1774 tablet_replica.cc:333] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.658380 1774 raft_consensus.cc:2243] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.658567 1774 raft_consensus.cc:2272] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.659174 1774 ts_tablet_manager.cc:1916] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.664476 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_737f344a623f459aa91cedb039b78ffe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.665258 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_737f344a623f459aa91cedb039b78ffe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.667142 1774 ts_tablet_manager.cc:1929] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.667239 1774 log.cc:1199] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/6424b2347ef642feb38a9596f2d2be1d
I20251024 19:48:22.667587 1774 ts_tablet_manager.cc:1950] T 6424b2347ef642feb38a9596f2d2be1d P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:22.668241 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 973) took 1089 ms. Trace:
I20251024 19:48:22.668344 1774 rpcz_store.cc:276] 1024 19:48:21.578821 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.578893 (+ 72us) service_pool.cc:225] Handling call
1024 19:48:22.658106 (+1079213us) ts_tablet_manager.cc:1151] Deleting tablet 6424b2347ef642feb38a9596f2d2be1d
1024 19:48:22.658118 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.663272 (+ 5154us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.667073 (+ 3801us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.668224 (+ 1151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":956460}
I20251024 19:48:22.668725 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 6424b2347ef642feb38a9596f2d2be1d (table default.Table_da1c2655e0de479e91cf0a21e1a728b5 [id=42ffad851c714cb28e672b448efb33bd]) successfully deleted
I20251024 19:48:22.668989 1774 tablet_replica.cc:333] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.669164 1774 raft_consensus.cc:2243] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.669297 1774 raft_consensus.cc:2272] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.670092 1774 ts_tablet_manager.cc:1916] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.671897 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0c171a897bcc47e5a625cfbb07056e74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.672508 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c171a897bcc47e5a625cfbb07056e74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.671823 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2691f5c07d014467892d86561d71800b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.673509 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2691f5c07d014467892d86561d71800b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.681519 1774 ts_tablet_manager.cc:1929] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.681648 1774 log.cc:1199] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/129debb55f4a4b7295a8ad88e55ebc83
I20251024 19:48:22.682014 1774 ts_tablet_manager.cc:1950] T 129debb55f4a4b7295a8ad88e55ebc83 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:22.683161 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 129debb55f4a4b7295a8ad88e55ebc83 (table default.Table_c777799df90e491992909e8a45205a06 [id=d11ca7b446a44dd28c9c243a6d7787e6]) successfully deleted
W20251024 19:48:22.690330 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 703441e1af8e4356afd22ef68dac87a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 703441e1af8e4356afd22ef68dac87a2 already in progress: creating tablet
I20251024 19:48:22.690514 3296 raft_consensus.cc:493] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.690649 3296 raft_consensus.cc:515] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.691279 3296 leader_election.cc:290] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:22.691483 1774 tablet_replica.cc:333] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.691682 1774 raft_consensus.cc:2243] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.691828 1774 raft_consensus.cc:2272] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.692368 1774 ts_tablet_manager.cc:1916] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:22.692929 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 68d85226ac5b4e4ab22a49dc419e375a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68d85226ac5b4e4ab22a49dc419e375a already in progress: creating tablet
W20251024 19:48:22.693182 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4ddce416119e4f08b81be4b5e6760144 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ddce416119e4f08b81be4b5e6760144 already in progress: creating tablet
I20251024 19:48:22.693918 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3d80885fcc642b8a75da30cd4abecbd" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:22.705714 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 61577.6us
I20251024 19:48:22.707792 1774 ts_tablet_manager.cc:1929] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.707916 1774 log.cc:1199] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/edb49884389c4dba957f0ee5af4ae0e4
I20251024 19:48:22.644232 3361 raft_consensus.cc:493] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.713727 3361 raft_consensus.cc:515] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.714378 3361 leader_election.cc:290] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:22.714612 1776 rpcz_store.cc:276] 1024 19:48:20.090859 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.090927 (+ 68us) service_pool.cc:225] Handling call
1024 19:48:22.633903 (+2542976us) ts_tablet_manager.cc:1151] Deleting tablet 8d1b6eaa353c4284a2b114b42437272c
1024 19:48:22.633916 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.634019 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2542784}
I20251024 19:48:22.715171 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1025) took 2622 ms. Trace:
I20251024 19:48:22.715495 1776 rpcz_store.cc:276] 1024 19:48:20.092876 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.092926 (+ 50us) service_pool.cc:225] Handling call
1024 19:48:22.714959 (+2622033us) ts_tablet_manager.cc:1151] Deleting tablet 3ef6d244c498452ca79cd6c6d5c003df
1024 19:48:22.714988 (+ 29us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.715157 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2621884}
I20251024 19:48:22.715914 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1026) took 2620 ms. Trace:
I20251024 19:48:22.716058 1776 rpcz_store.cc:276] 1024 19:48:20.095693 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.095749 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:22.715806 (+2620057us) ts_tablet_manager.cc:1151] Deleting tablet 9287f340fb554904986f6c18b7356701
1024 19:48:22.715815 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.715899 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2619893}
I20251024 19:48:22.716504 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1027) took 2620 ms. Trace:
I20251024 19:48:22.716635 1776 rpcz_store.cc:276] 1024 19:48:20.096047 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.096234 (+ 187us) service_pool.cc:225] Handling call
1024 19:48:22.716338 (+2620104us) ts_tablet_manager.cc:1151] Deleting tablet 3c708051e9754fe8afc45154b0a4f30c
1024 19:48:22.716346 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.716486 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2619933}
I20251024 19:48:22.717218 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1028) took 2621 ms. Trace:
I20251024 19:48:22.717504 1776 rpcz_store.cc:276] 1024 19:48:20.096165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.096235 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:22.717105 (+2620870us) ts_tablet_manager.cc:1151] Deleting tablet 3a712f52cd114e9bb07c2717e5d728fd
1024 19:48:22.717117 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.717200 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2620362}
I20251024 19:48:22.719863 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 6424b2347ef642feb38a9596f2d2be1d (table default.Table_da1c2655e0de479e91cf0a21e1a728b5 [id=42ffad851c714cb28e672b448efb33bd]) successfully deleted
I20251024 19:48:22.720074 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (table default.Table_18e7285b7df44399ab3be1dfc86eb91f [id=14bda7f9324f44aa9cb87a5f129039be]). Expected version 2 got 0
I20251024 19:48:22.720171 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 50cea450eda9485daba40bf875a1b90c (table default.Table_ac4ce453efea4a268f70380f9ed99812 [id=bfd41da632b94dcf90b0e0d230dc8fb5]) successfully deleted
W20251024 19:48:22.720471 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9a94c856db4141cfb7ada7515d5b6e9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a94c856db4141cfb7ada7515d5b6e9e already in progress: creating tablet
W20251024 19:48:22.720757 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f6df45dfab124358b8669bb4f01a6dd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6df45dfab124358b8669bb4f01a6dd5 already in progress: creating tablet
I20251024 19:48:22.721052 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet e03c488d0f264684b0897a81a04d7466 (table default.Table_97ae72e3d285443a8435906795593d52 [id=5c054cd9fe2c4684854e0da6e68bc7d8]) successfully deleted
W20251024 19:48:22.721346 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 8d1b6eaa353c4284a2b114b42437272c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1b6eaa353c4284a2b114b42437272c already in progress: creating tablet
I20251024 19:48:22.721653 1774 ts_tablet_manager.cc:1950] T edb49884389c4dba957f0ee5af4ae0e4 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:48:22.721657 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3ef6d244c498452ca79cd6c6d5c003df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ef6d244c498452ca79cd6c6d5c003df already in progress: creating tablet
I20251024 19:48:22.722766 1774 tablet_replica.cc:333] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
W20251024 19:48:22.721923 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9287f340fb554904986f6c18b7356701 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9287f340fb554904986f6c18b7356701 already in progress: creating tablet
I20251024 19:48:22.722978 1774 raft_consensus.cc:2243] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:22.723006 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3c708051e9754fe8afc45154b0a4f30c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c708051e9754fe8afc45154b0a4f30c already in progress: creating tablet
I20251024 19:48:22.723119 1774 raft_consensus.cc:2272] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.724611 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet f3d80885fcc642b8a75da30cd4abecbd with delete_type TABLET_DATA_DELETED (replaced by tablet 780442b310e6409498e6bbe4256280a4) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.724659 1183 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 4100abc9ff12471e9a4959b6b24f7e99 (table default.Table_da24358ca9544a0e805bb937715888f9 [id=0e3c3f6ff360497c93fac944f351702c]). Expected version 1 got 0
I20251024 19:48:22.725761 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 28005301f9de48e5826f46c79aeb5fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.726029 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet edb49884389c4dba957f0ee5af4ae0e4 (table default.Table_07aae1757a3444a4bbb8d729d6f08a1d [id=b410120363fe4ba5907be2c8793b5e8c]) successfully deleted
W20251024 19:48:22.726263 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet c89fa8932dbb4f4fbf7fd9f57bff27c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c89fa8932dbb4f4fbf7fd9f57bff27c3 already in progress: creating tablet
I20251024 19:48:22.726742 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_562481ab77914c9da01e8f0c63e93035" } new_table_name: "default.Table_382d1f0a0b8e4509ab298430d6b095fb" modify_external_catalogs: true
I20251024 19:48:22.728699 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1037786a8dff4c57b157b251d8f39cfb" } new_table_name: "default.Table_39850b521efa4b7abf562d8460ac1e49" modify_external_catalogs: true
I20251024 19:48:22.731227 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet d840888183ec4ef7bb245363c5bac322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:22.731491 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 7be9290daabb404f955f05922d897fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:22.731691 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 73278db2097d4001a538d62dacb97c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:22.731761 1774 ts_tablet_manager.cc:1916] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.733494 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4e14a82420f24157a36df44d050b88bf
I20251024 19:48:22.735702 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e14a82420f24157a36df44d050b88bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.736232 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e14a82420f24157a36df44d050b88bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:22.738050 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 780442b310e6409498e6bbe4256280a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:22.743546 1774 ts_tablet_manager.cc:1929] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.743644 1774 log.cc:1199] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/ee68243b4a664575b679950603bb250e
I20251024 19:48:22.743985 1774 ts_tablet_manager.cc:1950] T ee68243b4a664575b679950603bb250e P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:22.744972 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet ee68243b4a664575b679950603bb250e (table default.Table_605117e2eb734e6b8dc34ceb4b5b5b71 [id=38b6e1273e1a43bd8b12604db5371677]) successfully deleted
W20251024 19:48:22.746532 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef837960a0544202834a0ca509089b29 consensus state has no leader
I20251024 19:48:22.748114 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e19e1939816b40ddb5131877638d430b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:22.748822 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7074697285141e3a62f8f372d86a3c9 consensus state has no leader
W20251024 19:48:22.748992 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e19e1939816b40ddb5131877638d430b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.749214 1774 tablet_replica.cc:333] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.749253 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fd2e2bba8e754ca3bdf8e7f36b01a40f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:22.749402 1774 raft_consensus.cc:2243] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:22.749961 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd2e2bba8e754ca3bdf8e7f36b01a40f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.749552 1774 raft_consensus.cc:2272] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.750669 1774 ts_tablet_manager.cc:1916] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.752116 1768 tablet_bootstrap.cc:492] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:22.752151 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 57f522a825264ea7b215304dbb7aa2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:22.752318 1768 ts_tablet_manager.cc:1403] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.126s user 0.002s sys 0.038s
I20251024 19:48:22.753140 1768 raft_consensus.cc:359] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.753293 1768 raft_consensus.cc:385] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.753340 1768 raft_consensus.cc:740] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.753547 1768 consensus_queue.cc:260] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.754082 1768 ts_tablet_manager.cc:1434] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:22.754331 1768 tablet_bootstrap.cc:492] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:22.758015 1776 tablet_replica.cc:333] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:48:22.758045 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3a712f52cd114e9bb07c2717e5d728fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a712f52cd114e9bb07c2717e5d728fd already in progress: creating tablet
I20251024 19:48:22.758296 1776 raft_consensus.cc:2243] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:22.758646 1776 raft_consensus.cc:2272] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.721607 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3d80885fcc642b8a75da30cd4abecbd" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:22.722265 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d840888183ec4ef7bb245363c5bac322" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:22.723639 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet d2a68bb1dc6c483388784b9bb19ce065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:22.759169 1776 ts_tablet_manager.cc:1916] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.759737 1766 tablet_bootstrap.cc:492] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:22.762058 1766 ts_tablet_manager.cc:1403] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.406s user 0.006s sys 0.002s
I20251024 19:48:22.723924 3360 raft_consensus.cc:493] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.738647 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e14a82420f24157a36df44d050b88bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:22.764249 3360 raft_consensus.cc:515] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.764977 3360 leader_election.cc:290] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:22.765789 1774 ts_tablet_manager.cc:1929] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:22.765897 1774 log.cc:1199] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/3e6db58fac5e4af9b27bde6c223aa845
I20251024 19:48:22.765229 1305 leader_election.cc:304] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:22.765242 1766 raft_consensus.cc:359] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.766474 1305 leader_election.cc:304] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:22.766551 1766 raft_consensus.cc:385] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.766759 1766 raft_consensus.cc:740] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.744299 3361 raft_consensus.cc:493] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.766868 3296 raft_consensus.cc:2749] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:22.767004 3296 raft_consensus.cc:2749] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:22.766948 3361 raft_consensus.cc:515] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:22.701918 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 211476us
I20251024 19:48:22.767383 1768 tablet_bootstrap.cc:654] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.767445 3361 leader_election.cc:290] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:22.767846 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "385e71e672154e399f14321eac0d8625" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:22.768702 1766 consensus_queue.cc:260] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.777017 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "385e71e672154e399f14321eac0d8625" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:22.777268 1504 raft_consensus.cc:3060] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:22.777781 1178 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet b7074697285141e3a62f8f372d86a3c9 (table default.Table_39850b521efa4b7abf562d8460ac1e49 [id=231e82511d8142608ed6dd5f47f610fe], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet b7074697285141e3a62f8f372d86a3c9 consensus state has no leader
I20251024 19:48:22.780202 1504 raft_consensus.cc:2468] T 385e71e672154e399f14321eac0d8625 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:22.782068 1775 ts_tablet_manager.cc:1929] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:22.782246 1775 log.cc:1199] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/d2a68bb1dc6c483388784b9bb19ce065
I20251024 19:48:22.782466 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10891ff5bf0d4b93bcab9faa5f7346f1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:22.782649 1183 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet a1fbbba88a294adfbf97c0343d86698e (table default.Table_7bc8cd4a930b44dfafa473297875594d [id=a60c8222aa0248c091fab8dc34100847]). Expected version 2 got 0
I20251024 19:48:22.782785 1775 ts_tablet_manager.cc:1950] T d2a68bb1dc6c483388784b9bb19ce065 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:22.783433 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10891ff5bf0d4b93bcab9faa5f7346f1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:22.783661 1504 raft_consensus.cc:3060] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:22.784938 1774 ts_tablet_manager.cc:1950] T 3e6db58fac5e4af9b27bde6c223aa845 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:22.785876 1774 tablet_replica.cc:333] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.786057 1774 raft_consensus.cc:2243] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.786140 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet ab488e4971874a16b2e052c84c250777 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:22.786214 1774 raft_consensus.cc:2272] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.795313 1504 raft_consensus.cc:2468] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:22.796376 1247 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4e14a82420f24157a36df44d050b88bf
I20251024 19:48:22.797911 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e14a82420f24157a36df44d050b88bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:22.799642 1476 tablet_service.cc:1505] Processing CreateTablet for tablet f0c64bf1dfa240a29afe01ad5c73b97c (DEFAULT_TABLE table=default.Table_0fbe3a3cf6b9411495798ac85968de6d [id=5ee67de7ee194e349a6702263291cfaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.800137 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0c64bf1dfa240a29afe01ad5c73b97c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.801040 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a3572e58f56043ca8b41124f7b194cee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:22.801702 1774 ts_tablet_manager.cc:1916] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:22.805208 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3572e58f56043ca8b41124f7b194cee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:22.805722 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1011) took 2765 ms. Trace:
I20251024 19:48:22.805936 1775 rpcz_store.cc:276] 1024 19:48:20.039962 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.040018 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:22.351206 (+2311188us) ts_tablet_manager.cc:1151] Deleting tablet d2a68bb1dc6c483388784b9bb19ce065
1024 19:48:22.351221 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.624763 (+273542us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.781946 (+157183us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.805518 (+ 23572us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2311000}
I20251024 19:48:22.806360 1775 tablet_replica.cc:333] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:22.806507 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet d2a68bb1dc6c483388784b9bb19ce065 (table default.Table_3be1c2c5954947969e6903a3f016d589 [id=0b304e4408b24fe797f4438b874e480f]) successfully deleted
I20251024 19:48:22.806640 1775 raft_consensus.cc:2243] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.806871 1775 raft_consensus.cc:2272] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.807618 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e14a82420f24157a36df44d050b88bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:22.808040 1615 tablet_service.cc:1505] Processing CreateTablet for tablet f0c64bf1dfa240a29afe01ad5c73b97c (DEFAULT_TABLE table=default.Table_0fbe3a3cf6b9411495798ac85968de6d [id=5ee67de7ee194e349a6702263291cfaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.808406 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0c64bf1dfa240a29afe01ad5c73b97c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.811970 1776 ts_tablet_manager.cc:1929] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.812098 1776 log.cc:1199] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/81a3de69a13c459b9b7c02761ab7261e
I20251024 19:48:22.812167 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0266cef4e2034303978d7456c47ea98c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:22.812358 1504 raft_consensus.cc:2468] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:22.812477 1776 ts_tablet_manager.cc:1950] T 81a3de69a13c459b9b7c02761ab7261e P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:22.813400 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65424b00ab9c454f9584048caba83e6a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:22.813638 1504 raft_consensus.cc:2468] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:22.814179 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 97ede13cce3249d19001294d1da815c0 (DEFAULT_TABLE table=default.Table_368605d650e9482682f6d88537e0f51a [id=a1e424505f6d4739906a12af9a1410a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.814584 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97ede13cce3249d19001294d1da815c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.815492 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0266cef4e2034303978d7456c47ea98c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:22.815788 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65424b00ab9c454f9584048caba83e6a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:22.816154 1364 raft_consensus.cc:3060] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:22.817888 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 97ede13cce3249d19001294d1da815c0 (DEFAULT_TABLE table=default.Table_368605d650e9482682f6d88537e0f51a [id=a1e424505f6d4739906a12af9a1410a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.818305 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97ede13cce3249d19001294d1da815c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.833242 1624 tablet_service.cc:1505] Processing CreateTablet for tablet a6a3e3530dcd4b74ad55620d2923e86b (DEFAULT_TABLE table=default.Table_9d66cdbfe5824468b9e6733962e5ec85 [id=37b6b7f115e3488db291ed5597802ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.833492 1487 tablet_service.cc:1505] Processing CreateTablet for tablet a6a3e3530dcd4b74ad55620d2923e86b (DEFAULT_TABLE table=default.Table_9d66cdbfe5824468b9e6733962e5ec85 [id=37b6b7f115e3488db291ed5597802ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.833655 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a3e3530dcd4b74ad55620d2923e86b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.833957 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a3e3530dcd4b74ad55620d2923e86b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:22.840724 1586 leader_election.cc:343] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:22.840911 1586 leader_election.cc:343] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:22.842033 1586 leader_election.cc:343] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:22.842223 1586 leader_election.cc:343] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:22.850829 3363 consensus_queue.cc:1048] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:22.851761 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eafde5a40df4fbfb03340b7bf791a2a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:22.852995 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eafde5a40df4fbfb03340b7bf791a2a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:22.853353 1504 raft_consensus.cc:2468] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:22.854372 1775 ts_tablet_manager.cc:1916] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.856046 1767 tablet_bootstrap.cc:492] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:22.856328 1767 ts_tablet_manager.cc:1403] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.303s user 0.012s sys 0.027s
I20251024 19:48:22.864351 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 4e14a82420f24157a36df44d050b88bf (DEFAULT_TABLE table=default.Table_72458dd9d8744a1ba775cfdcd2cc6e8e [id=575d0247e06943d1ab1ccd4c6d40ceb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.865218 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e14a82420f24157a36df44d050b88bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.866927 1483 tablet_service.cc:1505] Processing CreateTablet for tablet e5f327b229f94cedb2ac5e5b785e99df (DEFAULT_TABLE table=default.Table_58d5b87034ec4242b1ab84dabf05f01b [id=d410e9b3180f4dd4adbeee42b838ae1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.867369 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5f327b229f94cedb2ac5e5b785e99df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.869267 1366 raft_consensus.cc:3060] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:22.870666 1624 tablet_service.cc:1505] Processing CreateTablet for tablet e5f327b229f94cedb2ac5e5b785e99df (DEFAULT_TABLE table=default.Table_58d5b87034ec4242b1ab84dabf05f01b [id=d410e9b3180f4dd4adbeee42b838ae1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.871081 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5f327b229f94cedb2ac5e5b785e99df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.873867 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1030) took 2668 ms. Trace:
I20251024 19:48:22.873998 1776 rpcz_store.cc:276] 1024 19:48:20.205640 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.205701 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:22.757959 (+2552258us) ts_tablet_manager.cc:1151] Deleting tablet 81a3de69a13c459b9b7c02761ab7261e
1024 19:48:22.757971 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.769195 (+ 11224us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.811861 (+ 42666us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:22.873850 (+ 61989us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2552061}
I20251024 19:48:22.874586 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1031) took 2667 ms. Trace:
I20251024 19:48:22.874738 1776 rpcz_store.cc:276] 1024 19:48:20.206656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.206705 (+ 49us) service_pool.cc:225] Handling call
1024 19:48:22.874446 (+2667741us) ts_tablet_manager.cc:1151] Deleting tablet 6b1a3523954448de93357510e6c1ec4e
1024 19:48:22.874459 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.874568 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2667555}
I20251024 19:48:22.875068 1776 tablet_replica.cc:333] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:22.875401 1776 raft_consensus.cc:2243] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:22.875763 1776 raft_consensus.cc:2272] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.876153 1364 raft_consensus.cc:3060] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:22.876708 1586 leader_election.cc:304] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:22.877060 1586 leader_election.cc:304] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:22.877676 2923 raft_consensus.cc:2804] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:22.877784 2923 raft_consensus.cc:697] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:22.877652 3137 raft_consensus.cc:2804] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:22.878595 3137 raft_consensus.cc:697] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:22.879176 3137 consensus_queue.cc:237] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:22.877993 2923 consensus_queue.cc:237] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.886296 1774 ts_tablet_manager.cc:1929] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:22.888782 1776 ts_tablet_manager.cc:1916] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.895628 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 9b079160403140599f065db37739b145 (DEFAULT_TABLE table=default.Table_f7dee67353c1481485de2f883d0bf7bf [id=4efade1dc2b84594b6208eec3bb308ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.896088 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b079160403140599f065db37739b145. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.897223 2923 raft_consensus.cc:493] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.897351 2923 raft_consensus.cc:515] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.897980 2923 leader_election.cc:290] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:22.902004 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 81a3de69a13c459b9b7c02761ab7261e (table default.Table_b5a66c4e9cd8408782690b66cdb87473 [id=3d82ad2f9e184043aed4ec0b05df724f]) successfully deleted
W20251024 19:48:22.902679 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 6b1a3523954448de93357510e6c1ec4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b1a3523954448de93357510e6c1ec4e already in progress: creating tablet
I20251024 19:48:22.903707 1613 tablet_service.cc:1505] Processing CreateTablet for tablet bc9d3d0d0047477287fa8c9d7fe7dd63 (DEFAULT_TABLE table=default.Table_4ecf5be7f6504214bc074337a9ddd17a [id=3c6a05dbb178489a98644e014ea3baf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.903755 1483 tablet_service.cc:1505] Processing CreateTablet for tablet bd2c43c6f497496d85cb35cc44f5ac8e (DEFAULT_TABLE table=default.Table_4d52c098fe604c2f885442d39f4238b7 [id=3a57bfedb2ae424fba4470fa161b3e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.904104 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9d3d0d0047477287fa8c9d7fe7dd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.904174 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd2c43c6f497496d85cb35cc44f5ac8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.905874 1616 tablet_service.cc:1505] Processing CreateTablet for tablet bd2c43c6f497496d85cb35cc44f5ac8e (DEFAULT_TABLE table=default.Table_4d52c098fe604c2f885442d39f4238b7 [id=3a57bfedb2ae424fba4470fa161b3e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.906214 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd2c43c6f497496d85cb35cc44f5ac8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.862380 1766 ts_tablet_manager.cc:1434] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.100s user 0.001s sys 0.001s
I20251024 19:48:22.865824 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 4e14a82420f24157a36df44d050b88bf (DEFAULT_TABLE table=default.Table_72458dd9d8744a1ba775cfdcd2cc6e8e [id=575d0247e06943d1ab1ccd4c6d40ceb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.866070 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 9b079160403140599f065db37739b145 (DEFAULT_TABLE table=default.Table_f7dee67353c1481485de2f883d0bf7bf [id=4efade1dc2b84594b6208eec3bb308ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.911916 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e14a82420f24157a36df44d050b88bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.912150 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b079160403140599f065db37739b145. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.919422 1774 log.cc:1199] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/3cf4596b333244cc9e9e35a128721e4a
I20251024 19:48:22.919952 1774 ts_tablet_manager.cc:1950] T 3cf4596b333244cc9e9e35a128721e4a P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:22.921312 1766 tablet_bootstrap.cc:492] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:22.921656 1774 tablet_replica.cc:333] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:22.921919 1774 raft_consensus.cc:2243] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:22.923218 1151 generic_service.cc:196] Checking for leaks (request via RPC)
I20251024 19:48:22.863284 1767 raft_consensus.cc:359] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.924813 1767 raft_consensus.cc:385] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.925679 1767 raft_consensus.cc:740] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.926222 1767 consensus_queue.cc:260] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.929060 1767 ts_tablet_manager.cc:1434] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.066s user 0.001s sys 0.001s
W20251024 19:48:22.937721 1305 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 171253us
I20251024 19:48:22.938391 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 3e6db58fac5e4af9b27bde6c223aa845 (table default.Table_78e6806b2ede4763843892de1ab015e3 [id=1ac4b7fed3f145fc9ef5268266aeb9f1]) successfully deleted
I20251024 19:48:22.939663 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet 59956dad06064024ab171631ef6c1dca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.939874 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e14a82420f24157a36df44d050b88bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.940414 1343 tablet_service.cc:1505] Processing CreateTablet for tablet f0c64bf1dfa240a29afe01ad5c73b97c (DEFAULT_TABLE table=default.Table_0fbe3a3cf6b9411495798ac85968de6d [id=5ee67de7ee194e349a6702263291cfaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.940840 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0c64bf1dfa240a29afe01ad5c73b97c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.940909 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 3cf4596b333244cc9e9e35a128721e4a (table default.Table_25adc9559f2b43e2a4adc92dfc0c7807 [id=60d23d1c07704c33bc979f7ba025b102]) successfully deleted
W20251024 19:48:22.941169 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet aa08555dbe1e4ad8968c534510eca9ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa08555dbe1e4ad8968c534510eca9ff already in progress: creating tablet
W20251024 19:48:22.941406 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet ce374bf694604250b3bbef15856ad01d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce374bf694604250b3bbef15856ad01d already in progress: creating tablet
W20251024 19:48:22.941717 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet d9367428428648f7ae07722ea09d70a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9367428428648f7ae07722ea09d70a7 already in progress: creating tablet
I20251024 19:48:22.949273 1352 tablet_service.cc:1505] Processing CreateTablet for tablet a6a3e3530dcd4b74ad55620d2923e86b (DEFAULT_TABLE table=default.Table_9d66cdbfe5824468b9e6733962e5ec85 [id=37b6b7f115e3488db291ed5597802ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.949441 1341 tablet_service.cc:1505] Processing CreateTablet for tablet 9b079160403140599f065db37739b145 (DEFAULT_TABLE table=default.Table_f7dee67353c1481485de2f883d0bf7bf [id=4efade1dc2b84594b6208eec3bb308ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.949859 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 4e14a82420f24157a36df44d050b88bf (DEFAULT_TABLE table=default.Table_72458dd9d8744a1ba775cfdcd2cc6e8e [id=575d0247e06943d1ab1ccd4c6d40ceb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.950176 1346 tablet_service.cc:1505] Processing CreateTablet for tablet e5f327b229f94cedb2ac5e5b785e99df (DEFAULT_TABLE table=default.Table_58d5b87034ec4242b1ab84dabf05f01b [id=d410e9b3180f4dd4adbeee42b838ae1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.950268 1349 tablet_service.cc:1505] Processing CreateTablet for tablet bc9d3d0d0047477287fa8c9d7fe7dd63 (DEFAULT_TABLE table=default.Table_4ecf5be7f6504214bc074337a9ddd17a [id=3c6a05dbb178489a98644e014ea3baf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.951053 1336 tablet_service.cc:1505] Processing CreateTablet for tablet bd2c43c6f497496d85cb35cc44f5ac8e (DEFAULT_TABLE table=default.Table_4d52c098fe604c2f885442d39f4238b7 [id=3a57bfedb2ae424fba4470fa161b3e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.953400 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 97ede13cce3249d19001294d1da815c0 (DEFAULT_TABLE table=default.Table_368605d650e9482682f6d88537e0f51a [id=a1e424505f6d4739906a12af9a1410a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.877650 1586 leader_election.cc:304] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:22.959893 1768 tablet_bootstrap.cc:492] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:22.960017 1768 ts_tablet_manager.cc:1403] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.206s user 0.024s sys 0.035s
W20251024 19:48:22.959895 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 82193.7us
I20251024 19:48:22.960449 1586 leader_election.cc:304] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:22.960657 1767 tablet_bootstrap.cc:492] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:22.961059 1768 raft_consensus.cc:359] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.961220 1768 raft_consensus.cc:385] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:22.961301 1768 raft_consensus.cc:740] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:22.961434 2923 raft_consensus.cc:2804] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:22.961542 2923 raft_consensus.cc:493] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:22.961481 1768 consensus_queue.cc:260] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.961663 2923 raft_consensus.cc:3060] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:22.961946 1768 ts_tablet_manager.cc:1434] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:22.962229 1768 tablet_bootstrap.cc:492] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:22.964298 2923 raft_consensus.cc:515] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.964625 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd2c43c6f497496d85cb35cc44f5ac8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.964833 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5f327b229f94cedb2ac5e5b785e99df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.964912 2923 leader_election.cc:290] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:22.965086 2923 raft_consensus.cc:2804] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:22.965176 2923 raft_consensus.cc:493] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:22.965255 2923 raft_consensus.cc:3060] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:22.965678 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a3e3530dcd4b74ad55620d2923e86b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.965937 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b079160403140599f065db37739b145. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.968036 1768 tablet_bootstrap.cc:654] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:22.970366 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9d3d0d0047477287fa8c9d7fe7dd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.971220 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e14a82420f24157a36df44d050b88bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.973240 3296 raft_consensus.cc:493] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.973393 3296 raft_consensus.cc:515] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.974152 3296 leader_election.cc:290] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:22.984266 1774 raft_consensus.cc:2272] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:22.985962 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97ede13cce3249d19001294d1da815c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.987035 2923 raft_consensus.cc:515] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.989207 2858 raft_consensus.cc:493] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:22.989379 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 24d5f5618e1947f79b81239baede8e1a (DEFAULT_TABLE table=default.Table_00523012c9d142bc81e566a6e515e07f [id=c4645844a34e47709f63c6aaa5d76f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.989784 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28005301f9de48e5826f46c79aeb5fe5" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:22.995002 2858 raft_consensus.cc:515] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:22.996027 1774 ts_tablet_manager.cc:1916] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:22.996646 2858 leader_election.cc:290] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:22.996811 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24d5f5618e1947f79b81239baede8e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:22.997548 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 24d5f5618e1947f79b81239baede8e1a (DEFAULT_TABLE table=default.Table_00523012c9d142bc81e566a6e515e07f [id=c4645844a34e47709f63c6aaa5d76f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.998087 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24d5f5618e1947f79b81239baede8e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.006845 1487 tablet_service.cc:1505] Processing CreateTablet for tablet ab16a80ca7224dc5b29fb87cf7ac68f0 (DEFAULT_TABLE table=default.Table_a1a3ffe843e4470aa847383e6f72077b [id=6a6b8eed1f364cee9eafec64e7353d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.007604 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab16a80ca7224dc5b29fb87cf7ac68f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.009935 2858 raft_consensus.cc:493] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:23.010314 2858 raft_consensus.cc:515] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:23.011260 2858 leader_election.cc:290] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:23.016258 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 1da15e7159ca45f79e1bb5b51453a0eb (DEFAULT_TABLE table=default.Table_038012ef539b428f9984af28f5dec4c8 [id=d7785be5e6694643a6e1e3872a40c71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.017016 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da15e7159ca45f79e1bb5b51453a0eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.017428 1341 tablet_service.cc:1505] Processing CreateTablet for tablet ab16a80ca7224dc5b29fb87cf7ac68f0 (DEFAULT_TABLE table=default.Table_a1a3ffe843e4470aa847383e6f72077b [id=6a6b8eed1f364cee9eafec64e7353d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.017860 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab16a80ca7224dc5b29fb87cf7ac68f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.018982 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 4efaca8915ee43c6bd039667b86331f5 (DEFAULT_TABLE table=default.Table_ee30e626b58d40d294af6f355ea32931 [id=e3d8f221d2954be0ab807bfe14b41829]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.019521 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efaca8915ee43c6bd039667b86331f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.022208 1475 tablet_service.cc:1505] Processing CreateTablet for tablet fe1a3db8b6bf4b69bf6269d07a918ea0 (DEFAULT_TABLE table=default.Table_b8596dd14d1a4384a3aa199e688330af [id=82eec0518fc14654b391b75cbf5f1634]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.022652 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe1a3db8b6bf4b69bf6269d07a918ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.026052 1479 tablet_service.cc:1505] Processing CreateTablet for tablet b5ab87c63f364083839ca64d79e56fe9 (DEFAULT_TABLE table=default.Table_ae6a9de86a02421a921a5552af7a1e9c [id=3045997a6996491da02ec5cfb101cb87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.026499 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5ab87c63f364083839ca64d79e56fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.029821 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 80688d7b8a394a2a910c4b8910fe4cf2
I20251024 19:48:23.030182 1477 tablet_service.cc:1505] Processing CreateTablet for tablet da17f70abb374521ac092269897b80c8 (DEFAULT_TABLE table=default.Table_0ba3b1e96f494fb0821fc035f4b67162 [id=0f592041e57644c0b545803be6d1e7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.030633 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da17f70abb374521ac092269897b80c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.031946 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e517c79e1029452c8529bc3aef0c8338"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.032593 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e517c79e1029452c8529bc3aef0c8338 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.032619 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 83d34ec47c2845058e870c72cc832e9e
I20251024 19:48:23.034281 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 80688d7b8a394a2a910c4b8910fe4cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:23.034922 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 83d34ec47c2845058e870c72cc832e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:23.035779 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9198321bcc9e4e47a4c2a5442b6bc850
I20251024 19:48:23.037910 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9b7ab1bf5b9549afb150802a209527eb
I20251024 19:48:23.038475 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 9198321bcc9e4e47a4c2a5442b6bc850 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:23.039417 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b7ab1bf5b9549afb150802a209527eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:23.040354 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_73c2345bb48f422ba47714dc6770bc3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.040968 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73c2345bb48f422ba47714dc6770bc3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.042393 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_80225fc48e434efc981690a21c61cd00" } new_table_name: "default.Table_6ef65af7e15c4360b7808f7dd020727b" modify_external_catalogs: true
I20251024 19:48:23.046999 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 4efaca8915ee43c6bd039667b86331f5 (DEFAULT_TABLE table=default.Table_ee30e626b58d40d294af6f355ea32931 [id=e3d8f221d2954be0ab807bfe14b41829]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.047549 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efaca8915ee43c6bd039667b86331f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.047644 1774 ts_tablet_manager.cc:1929] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:23.048012 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7610674d2b04400c8e953eb393f066e7" } new_table_name: "default.Table_a0a1640847fd436a959bbde7d4215fc6" modify_external_catalogs: true
I20251024 19:48:23.048867 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_498e11a28046450cbcd95b82f686f7de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:23.049528 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 2b119e8ceae04852964513ebfa7aa31f (DEFAULT_TABLE table=default.Table_af70ad0ef8864f3a83c8376e84e7ddd3 [id=d0e3d80b2b7643d4a48fc48e4dd4b446]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.049918 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b119e8ceae04852964513ebfa7aa31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.051258 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4834ef486a9b47a9976759474b0b2ca1" } new_table_name: "default.Table_159163726a8743d6a8e6ee5e5d68c082" modify_external_catalogs: true
I20251024 19:48:23.048947 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet edb49884389c4dba957f0ee5af4ae0e4 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:23.053663 1774 log.cc:1199] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/9a1031d785b646b095ece9be666d2a77
I20251024 19:48:23.057888 1479 tablet_service.cc:1505] Processing CreateTablet for tablet bec3806757214c869cf1c4cecd7cc26e (DEFAULT_TABLE table=default.Table_40bc5f8e5a2d4aeeb9ce677ed9dd0ccb [id=66078b0ffd8a4b249244ecc682de51de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.058229 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bec3806757214c869cf1c4cecd7cc26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.058590 1774 ts_tablet_manager.cc:1950] T 9a1031d785b646b095ece9be666d2a77 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:23.059208 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_69b9df2416fe4680b34e9b399ef39216"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.060285 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69b9df2416fe4680b34e9b399ef39216 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.060346 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 2d01572e51694d9bb58c3a59d4205102 (table default.Table_bf15d3c700774490b1860d3d8865a95e [id=2ea13e78137d47429d7cbe374d2113ef]). Expected version 1 got 0
W20251024 19:48:23.059747 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_498e11a28046450cbcd95b82f686f7de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.060657 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (table default.Table_18e7285b7df44399ab3be1dfc86eb91f [id=14bda7f9324f44aa9cb87a5f129039be]). Expected version 2 got 0
I20251024 19:48:23.060982 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 1999b751e9344dcf81d26226c0589c8f (table default.Table_9147a98b3bf9483ea7b7a6f17130d4fc [id=d743705c21da4b89abedf2eb34ddbe09]). Expected version 1 got 0
I20251024 19:48:23.061096 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 1da15e7159ca45f79e1bb5b51453a0eb (DEFAULT_TABLE table=default.Table_038012ef539b428f9984af28f5dec4c8 [id=d7785be5e6694643a6e1e3872a40c71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.066524 1774 tablet_replica.cc:333] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:23.068591 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f30acab987914de4a2372f8aeec08030" } new_table_name: "default.Table_7060116dc4504d2fb09b13668396d97a" modify_external_catalogs: true
I20251024 19:48:23.069844 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da15e7159ca45f79e1bb5b51453a0eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.069932 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_12fe00934de64b41af4f5fa5980dd476" } modify_external_catalogs: true
I20251024 19:48:23.070116 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_12fe00934de64b41af4f5fa5980dd476" } modify_external_catalogs: true
I20251024 19:48:23.070413 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f5c4c9448ed5406cbc1c615cdcca8781" } new_table_name: "default.Table_b952a3ced9b94e19bcee920f864b326c" modify_external_catalogs: true
I20251024 19:48:23.070888 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 24ff2839ff1e48d5a2f84890ec25c75b (DEFAULT_TABLE table=default.Table_2f3f9f0797ae454eb0fd2975c56cadfd [id=e76a313dfab24c94aca3bae5a9426e7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.071306 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ff2839ff1e48d5a2f84890ec25c75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.068996 1774 raft_consensus.cc:2243] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:23.073671 1476 tablet_service.cc:1505] Processing CreateTablet for tablet bc9d3d0d0047477287fa8c9d7fe7dd63 (DEFAULT_TABLE table=default.Table_4ecf5be7f6504214bc074337a9ddd17a [id=3c6a05dbb178489a98644e014ea3baf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.074002 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9d3d0d0047477287fa8c9d7fe7dd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.073984 1774 raft_consensus.cc:2272] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:23.074059 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b39ea1cdfd63459c83c0f471a6c57015"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.074609 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b39ea1cdfd63459c83c0f471a6c57015 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.075104 1774 ts_tablet_manager.cc:1916] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:23.076491 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 2b32adf2d1db4cee8d22962b4af410ce (DEFAULT_TABLE table=default.Table_dd0f43494b6b45c1b34861a14440b2eb [id=fa92350ddd804229acc28b3200ed5b5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.076886 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b32adf2d1db4cee8d22962b4af410ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.079069 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2ead137126594dc39864dfac900057d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.079576 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ead137126594dc39864dfac900057d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.083719 1352 tablet_service.cc:1505] Processing CreateTablet for tablet fe1a3db8b6bf4b69bf6269d07a918ea0 (DEFAULT_TABLE table=default.Table_b8596dd14d1a4384a3aa199e688330af [id=82eec0518fc14654b391b75cbf5f1634]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.084132 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe1a3db8b6bf4b69bf6269d07a918ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.084074 1473 tablet_service.cc:1505] Processing CreateTablet for tablet dd42524522f24bad9120f5091ac684d5 (DEFAULT_TABLE table=default.Table_6237c63465cc4c8daa9136ba2d3a39fc [id=cca9386aa9f94dba969810eb36299f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.084442 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd42524522f24bad9120f5091ac684d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:23.087707 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 529580599bfa4800973b8540ab13d22a consensus state has no leader
I20251024 19:48:23.088274 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a316e0ba22744f2c9b7431da3dbed677"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.088956 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a316e0ba22744f2c9b7431da3dbed677 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:23.093760 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c084db163d874639b4385f41e9efaa18 consensus state has no leader
I20251024 19:48:23.094722 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ade70e87a32546219c5f6171be35ec6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:23.095177 1766 tablet_bootstrap.cc:654] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:23.095441 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ade70e87a32546219c5f6171be35ec6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.097432 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 06212ab82546412d9412415159932179 (DEFAULT_TABLE table=default.Table_4bd52fc39eef4537bafea34b959fa79c [id=c9f80c478640476aafebf901debe39b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.097844 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06212ab82546412d9412415159932179. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.103534 1346 tablet_service.cc:1505] Processing CreateTablet for tablet b5ab87c63f364083839ca64d79e56fe9 (DEFAULT_TABLE table=default.Table_ae6a9de86a02421a921a5552af7a1e9c [id=3045997a6996491da02ec5cfb101cb87]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:23.103739 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 593f5600e4274fce97e7bd33f025569f consensus state has no leader
I20251024 19:48:23.103956 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5ab87c63f364083839ca64d79e56fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.104095 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet b24a3a820c9b496dbf6955b399c6ea8c from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:23.109766 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 6292d44b4a4b4458b784334afc4ed28e (DEFAULT_TABLE table=default.Table_2def15e9c34c4a1ca47b1600a926f7c2 [id=0312537905ad4d708e581ba2fb357707]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.110183 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6292d44b4a4b4458b784334afc4ed28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.110867 1774 ts_tablet_manager.cc:1929] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:23.111025 1774 log.cc:1199] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/50cea450eda9485daba40bf875a1b90c
I20251024 19:48:23.111455 1774 ts_tablet_manager.cc:1950] T 50cea450eda9485daba40bf875a1b90c P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:23.111897 1774 tablet_replica.cc:333] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:23.112103 1774 raft_consensus.cc:2243] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:23.112274 1774 raft_consensus.cc:2272] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:23.114168 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet 80688d7b8a394a2a910c4b8910fe4cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:23.114334 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 9198321bcc9e4e47a4c2a5442b6bc850 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:23.114336 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet 83d34ec47c2845058e870c72cc832e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:23.114646 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b7ab1bf5b9549afb150802a209527eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:23.114902 1349 tablet_service.cc:1505] Processing CreateTablet for tablet da17f70abb374521ac092269897b80c8 (DEFAULT_TABLE table=default.Table_0ba3b1e96f494fb0821fc035f4b67162 [id=0f592041e57644c0b545803be6d1e7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.115108 1346 tablet_service.cc:1505] Processing CreateTablet for tablet bec3806757214c869cf1c4cecd7cc26e (DEFAULT_TABLE table=default.Table_40bc5f8e5a2d4aeeb9ce677ed9dd0ccb [id=66078b0ffd8a4b249244ecc682de51de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.115417 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da17f70abb374521ac092269897b80c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.115617 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 2b119e8ceae04852964513ebfa7aa31f (DEFAULT_TABLE table=default.Table_af70ad0ef8864f3a83c8376e84e7ddd3 [id=d0e3d80b2b7643d4a48fc48e4dd4b446]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.116020 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b119e8ceae04852964513ebfa7aa31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.117295 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet 786fe2fd6fca407e83a76f74f96a8b16 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:23.122267 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bec3806757214c869cf1c4cecd7cc26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.127300 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet cb1f850b3e104707a53d540ad5752d4d
I20251024 19:48:23.128577 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet cb1f850b3e104707a53d540ad5752d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:23.128608 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 2b32adf2d1db4cee8d22962b4af410ce (DEFAULT_TABLE table=default.Table_dd0f43494b6b45c1b34861a14440b2eb [id=fa92350ddd804229acc28b3200ed5b5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.128861 1343 tablet_service.cc:1505] Processing CreateTablet for tablet dd42524522f24bad9120f5091ac684d5 (DEFAULT_TABLE table=default.Table_6237c63465cc4c8daa9136ba2d3a39fc [id=cca9386aa9f94dba969810eb36299f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.128996 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b32adf2d1db4cee8d22962b4af410ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.129220 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd42524522f24bad9120f5091ac684d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.130133 1483 tablet_service.cc:1505] Processing CreateTablet for tablet d27750aaf3c84f5eae2df5f4337063c3 (DEFAULT_TABLE table=default.Table_3b12fc2aa8564985980b459691c71555 [id=1ad6109902444b2ab657b1d7f0d86cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.130585 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27750aaf3c84f5eae2df5f4337063c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.133767 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 9026d9e801294dd2bfdb9c90e45a0990 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:15 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:22.987852 1586 leader_election.cc:304] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:22.994761 1616 tablet_service.cc:1505] Processing CreateTablet for tablet ab16a80ca7224dc5b29fb87cf7ac68f0 (DEFAULT_TABLE table=default.Table_a1a3ffe843e4470aa847383e6f72077b [id=6a6b8eed1f364cee9eafec64e7353d96]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:23.135237 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 147441us
I20251024 19:48:23.135567 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab16a80ca7224dc5b29fb87cf7ac68f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.135653 3137 raft_consensus.cc:2804] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:23.135877 3137 raft_consensus.cc:493] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:23.136039 3137 raft_consensus.cc:3060] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:23.136300 3363 consensus_queue.cc:1048] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:23.128629 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 24ff2839ff1e48d5a2f84890ec25c75b (DEFAULT_TABLE table=default.Table_2f3f9f0797ae454eb0fd2975c56cadfd [id=e76a313dfab24c94aca3bae5a9426e7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.136848 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ff2839ff1e48d5a2f84890ec25c75b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:23.137097 1586 consensus_peers.cc:597] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:23.137315 1586 leader_election.cc:343] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:22.995296 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 24d5f5618e1947f79b81239baede8e1a (DEFAULT_TABLE table=default.Table_00523012c9d142bc81e566a6e515e07f [id=c4645844a34e47709f63c6aaa5d76f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:22.996389 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab488e4971874a16b2e052c84c250777" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:23.138144 1586 consensus_peers.cc:597] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b20a6845ae1645c0b51f096d471d4ccc. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:23.138537 1444 leader_election.cc:343] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:23.139521 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24d5f5618e1947f79b81239baede8e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.140039 3137 raft_consensus.cc:515] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:23.140151 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3440f176bd814230b220b32586cc1208 consensus state has no leader
W20251024 19:48:23.140520 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57e1e21beee44847b1ebeb746f02c372 consensus state has no leader
I20251024 19:48:23.140695 3137 leader_election.cc:290] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:23.012326 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28005301f9de48e5826f46c79aeb5fe5" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:23.012585 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f522a825264ea7b215304dbb7aa2d5" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:23.013269 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 1da15e7159ca45f79e1bb5b51453a0eb (DEFAULT_TABLE table=default.Table_038012ef539b428f9984af28f5dec4c8 [id=d7785be5e6694643a6e1e3872a40c71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.142545 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da15e7159ca45f79e1bb5b51453a0eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.142647 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f965111ac60841dbb9e6ecc21ff3fec0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:23.142812 1508 raft_consensus.cc:2468] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:23.143325 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0266cef4e2034303978d7456c47ea98c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:23.143448 1507 raft_consensus.cc:3060] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:23.022339 1627 tablet_service.cc:1505] Processing CreateTablet for tablet b5ab87c63f364083839ca64d79e56fe9 (DEFAULT_TABLE table=default.Table_ae6a9de86a02421a921a5552af7a1e9c [id=3045997a6996491da02ec5cfb101cb87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.145925 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5ab87c63f364083839ca64d79e56fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.022522 1621 tablet_service.cc:1505] Processing CreateTablet for tablet fe1a3db8b6bf4b69bf6269d07a918ea0 (DEFAULT_TABLE table=default.Table_b8596dd14d1a4384a3aa199e688330af [id=82eec0518fc14654b391b75cbf5f1634]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.031858 1776 ts_tablet_manager.cc:1929] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:23.149369 1776 log.cc:1199] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/3e6db58fac5e4af9b27bde6c223aa845
I20251024 19:48:23.149461 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe1a3db8b6bf4b69bf6269d07a918ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.149938 1776 ts_tablet_manager.cc:1950] T 3e6db58fac5e4af9b27bde6c223aa845 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:23.150906 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1032) took 2786 ms. Trace:
I20251024 19:48:23.151031 1776 rpcz_store.cc:276] 1024 19:48:20.364128 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.364194 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:22.875021 (+2510827us) ts_tablet_manager.cc:1151] Deleting tablet 3e6db58fac5e4af9b27bde6c223aa845
1024 19:48:22.875031 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:22.948792 (+ 73761us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.031738 (+ 82946us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.150831 (+119093us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2500816}
I20251024 19:48:23.151618 1776 tablet_replica.cc:333] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:23.151913 1776 raft_consensus.cc:2243] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:23.152362 1776 raft_consensus.cc:2272] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:23.152966 1776 ts_tablet_manager.cc:1916] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:23.156666 1507 raft_consensus.cc:2468] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
W20251024 19:48:23.157966 1444 leader_election.cc:343] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:23.066025 3360 raft_consensus.cc:493] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:23.098356 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 80688d7b8a394a2a910c4b8910fe4cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:23.098737 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 83d34ec47c2845058e870c72cc832e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:23.099300 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 9198321bcc9e4e47a4c2a5442b6bc850 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:23.099436 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b7ab1bf5b9549afb150802a209527eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:22 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:23.099440 1618 tablet_service.cc:1505] Processing CreateTablet for tablet bec3806757214c869cf1c4cecd7cc26e (DEFAULT_TABLE table=default.Table_40bc5f8e5a2d4aeeb9ce677ed9dd0ccb [id=66078b0ffd8a4b249244ecc682de51de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.160178 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bec3806757214c869cf1c4cecd7cc26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.099735 1628 tablet_service.cc:1552] Processing DeleteTablet for tablet 0266cef4e2034303978d7456c47ea98c with delete_type TABLET_DATA_DELETED (replaced by tablet 64325d89db714554866e74922d103e34) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:23.099972 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 65424b00ab9c454f9584048caba83e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:23.100486 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 2b119e8ceae04852964513ebfa7aa31f (DEFAULT_TABLE table=default.Table_af70ad0ef8864f3a83c8376e84e7ddd3 [id=d0e3d80b2b7643d4a48fc48e4dd4b446]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.164817 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b119e8ceae04852964513ebfa7aa31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.100967 1626 tablet_service.cc:1505] Processing CreateTablet for tablet 24ff2839ff1e48d5a2f84890ec25c75b (DEFAULT_TABLE table=default.Table_2f3f9f0797ae454eb0fd2975c56cadfd [id=e76a313dfab24c94aca3bae5a9426e7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.169556 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ff2839ff1e48d5a2f84890ec25c75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.101050 1624 tablet_service.cc:1505] Processing CreateTablet for tablet da17f70abb374521ac092269897b80c8 (DEFAULT_TABLE table=default.Table_0ba3b1e96f494fb0821fc035f4b67162 [id=0f592041e57644c0b545803be6d1e7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.174050 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da17f70abb374521ac092269897b80c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.101104 1625 tablet_service.cc:1505] Processing CreateTablet for tablet dd42524522f24bad9120f5091ac684d5 (DEFAULT_TABLE table=default.Table_6237c63465cc4c8daa9136ba2d3a39fc [id=cca9386aa9f94dba969810eb36299f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.178037 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd42524522f24bad9120f5091ac684d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.178372 1776 ts_tablet_manager.cc:1929] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:23.178566 1776 log.cc:1199] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/b20a6845ae1645c0b51f096d471d4ccc
I20251024 19:48:23.179109 1776 ts_tablet_manager.cc:1950] T b20a6845ae1645c0b51f096d471d4ccc P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:23.101382 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 2b32adf2d1db4cee8d22962b4af410ce (DEFAULT_TABLE table=default.Table_dd0f43494b6b45c1b34861a14440b2eb [id=fa92350ddd804229acc28b3200ed5b5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.182075 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b32adf2d1db4cee8d22962b4af410ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.184423 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b24a3a820c9b496dbf6955b399c6ea8c
I20251024 19:48:23.185204 1184 catalog_manager.cc:6719] ReplaceTablet: tablet b24a3a820c9b496dbf6955b399c6ea8c deleted and replaced by tablet 799780bf376843499c659df4cd2b4aa1
I20251024 19:48:23.101785 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 6292d44b4a4b4458b784334afc4ed28e (DEFAULT_TABLE table=default.Table_2def15e9c34c4a1ca47b1600a926f7c2 [id=0312537905ad4d708e581ba2fb357707]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.186034 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6292d44b4a4b4458b784334afc4ed28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.186115 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 9a1031d785b646b095ece9be666d2a77 (table default.Table_a1d09325af844d71b290d95b8a4aaf12 [id=b62ec7bcad7c41b1b946e9ebf2f4bf39]) successfully deleted
I20251024 19:48:23.186829 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 50cea450eda9485daba40bf875a1b90c (table default.Table_ac4ce453efea4a268f70380f9ed99812 [id=bfd41da632b94dcf90b0e0d230dc8fb5]) successfully deleted
I20251024 19:48:23.190779 2923 leader_election.cc:290] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:23.141757 1369 raft_consensus.cc:3060] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:23.146139 1365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0266cef4e2034303978d7456c47ea98c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:23.146891 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f965111ac60841dbb9e6ecc21ff3fec0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:23.184513 1350 tablet_service.cc:1505] Processing CreateTablet for tablet d27750aaf3c84f5eae2df5f4337063c3 (DEFAULT_TABLE table=default.Table_3b12fc2aa8564985980b459691c71555 [id=1ad6109902444b2ab657b1d7f0d86cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.184868 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet cb1f850b3e104707a53d540ad5752d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:23.195405 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27750aaf3c84f5eae2df5f4337063c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.195587 3360 raft_consensus.cc:515] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.201625 3360 leader_election.cc:290] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:23.185523 3374 raft_consensus.cc:493] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:23.185884 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab488e4971874a16b2e052c84c250777" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:23.186285 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f522a825264ea7b215304dbb7aa2d5" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:23.187108 1345 tablet_service.cc:1505] Processing CreateTablet for tablet f5c3483130e84b66b0e5e4784701ffbf (DEFAULT_TABLE table=default.Table_3e12a4891d024f54bb915e9f4dd0c60f [id=4852ccffc0284e1d83557455f4f1e9f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.202827 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c3483130e84b66b0e5e4784701ffbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.187745 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 19acf59898c8436e9d9fef98a7f07532 (DEFAULT_TABLE table=default.Table_dab0576c96374dd0b1cd1409c1594039 [id=f2f912a6c50e4fcdad32d3b096edbf97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.188135 1339 tablet_service.cc:1505] Processing CreateTablet for tablet e6fd158e8b0c4cc78cf7fd3b34d5f104 (DEFAULT_TABLE table=default.Table_67e6d3d8a4334247a1bf237d5936799b [id=b0c7cbd53bbd4b9fbaef10e6c8604cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.206248 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 6292d44b4a4b4458b784334afc4ed28e (DEFAULT_TABLE table=default.Table_2def15e9c34c4a1ca47b1600a926f7c2 [id=0312537905ad4d708e581ba2fb357707]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.189002 1333 tablet_service.cc:1552] Processing DeleteTablet for tablet b24a3a820c9b496dbf6955b399c6ea8c with delete_type TABLET_DATA_DELETED (replaced by tablet 799780bf376843499c659df4cd2b4aa1) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:23.189718 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 3a74d7661a5f445393b27e47744ecdf6 (DEFAULT_TABLE table=default.Table_9bcef35dbbd649ccbf09b1067e78bcc8 [id=c57bfdb496874a67a90e601c972818d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.207610 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1033) took 2805 ms. Trace:
I20251024 19:48:23.207779 1776 rpcz_store.cc:276] 1024 19:48:20.401694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.401773 (+ 79us) service_pool.cc:225] Handling call
1024 19:48:23.151391 (+2749618us) ts_tablet_manager.cc:1151] Deleting tablet b20a6845ae1645c0b51f096d471d4ccc
1024 19:48:23.151403 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:23.163941 (+ 12538us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.178173 (+ 14232us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.207558 (+ 29385us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2749386}
I20251024 19:48:23.208379 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1034) took 2787 ms. Trace:
I20251024 19:48:23.208587 1776 rpcz_store.cc:276] 1024 19:48:20.421246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.421448 (+ 202us) service_pool.cc:225] Handling call
1024 19:48:23.208139 (+2786691us) ts_tablet_manager.cc:1151] Deleting tablet fc187c8f34594f4fb37d8afe0855263b
1024 19:48:23.208152 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:23.208268 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2786327}
I20251024 19:48:23.208999 1776 tablet_replica.cc:333] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:23.209363 1776 raft_consensus.cc:2243] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:23.209933 2858 raft_consensus.cc:493] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:23.210076 2858 raft_consensus.cc:515] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.210638 2858 leader_election.cc:290] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:23.210680 1774 ts_tablet_manager.cc:1916] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:23.211325 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96a6e0a4882c42399fd07f06ac21aa3f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:23.211882 1444 leader_election.cc:343] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:23.216924 3374 raft_consensus.cc:515] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:23.219470 1308 leader_election.cc:343] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 9026d9e801294dd2bfdb9c90e45a0990
I20251024 19:48:23.219782 3374 leader_election.cc:290] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:23.229048 1776 raft_consensus.cc:2272] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:23.229974 1776 ts_tablet_manager.cc:1916] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:23.236953 1343 tablet_service.cc:1505] Processing CreateTablet for tablet 06212ab82546412d9412415159932179 (DEFAULT_TABLE table=default.Table_4bd52fc39eef4537bafea34b959fa79c [id=c9f80c478640476aafebf901debe39b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.238857 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_932da5af90f847b893732f7f8e1b982d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:23.239401 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2df2e83bbab54dcabedf78455d66e990" } modify_external_catalogs: true
W20251024 19:48:23.239622 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_932da5af90f847b893732f7f8e1b982d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.239683 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2df2e83bbab54dcabedf78455d66e990" } modify_external_catalogs: true
I20251024 19:48:23.242738 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2d74dcbcd9f249e5b10e1986cc0c0d18" } modify_external_catalogs: true
I20251024 19:48:23.242816 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8b808e9cdcbc4edd90f1804171f93ac4" } modify_external_catalogs: true
I20251024 19:48:23.242959 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2d74dcbcd9f249e5b10e1986cc0c0d18" } modify_external_catalogs: true
I20251024 19:48:23.243148 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8b808e9cdcbc4edd90f1804171f93ac4" } modify_external_catalogs: true
I20251024 19:48:23.245172 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 4efaca8915ee43c6bd039667b86331f5 (DEFAULT_TABLE table=default.Table_ee30e626b58d40d294af6f355ea32931 [id=e3d8f221d2954be0ab807bfe14b41829]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.246474 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efaca8915ee43c6bd039667b86331f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.248241 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d989d66502a645bc8cc2c0cef2bc0d35" } modify_external_catalogs: true
I20251024 19:48:23.255785 1768 tablet_bootstrap.cc:492] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:23.255694 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 06212ab82546412d9412415159932179 (DEFAULT_TABLE table=default.Table_4bd52fc39eef4537bafea34b959fa79c [id=c9f80c478640476aafebf901debe39b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.256193 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06212ab82546412d9412415159932179. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.256274 1768 ts_tablet_manager.cc:1403] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.294s user 0.027s sys 0.041s
I20251024 19:48:23.257332 1768 raft_consensus.cc:359] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.257666 1768 raft_consensus.cc:385] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:23.258054 1768 raft_consensus.cc:740] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:23.258463 1768 consensus_queue.cc:260] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.259146 1768 ts_tablet_manager.cc:1434] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251024 19:48:23.259744 1768 tablet_bootstrap.cc:492] T 385e71e672154e399f14321eac0d8625 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:23.263857 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet cb1f850b3e104707a53d540ad5752d4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:23.264441 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d989d66502a645bc8cc2c0cef2bc0d35" } modify_external_catalogs: true
I20251024 19:48:23.267836 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 830c8469331940fc85aaad58c9085e35
I20251024 19:48:23.270087 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 3e6db58fac5e4af9b27bde6c223aa845 (table default.Table_78e6806b2ede4763843892de1ab015e3 [id=1ac4b7fed3f145fc9ef5268266aeb9f1]) successfully deleted
I20251024 19:48:23.270817 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet b20a6845ae1645c0b51f096d471d4ccc (table default.Table_50e12cf5f39d4fa2983835ef0dbbc3d6 [id=201e5232763a4940a1288a56a695f71c]) successfully deleted
W20251024 19:48:23.271157 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet fc187c8f34594f4fb37d8afe0855263b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc187c8f34594f4fb37d8afe0855263b already in progress: creating tablet
I20251024 19:48:23.272297 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e3ba54a48d2c4fd594bbce560159e3aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.272869 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3ba54a48d2c4fd594bbce560159e3aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.274978 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9595525b6e594761ae90d7fcde6eaea4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:23.275451 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet b24a3a820c9b496dbf6955b399c6ea8c with delete_type TABLET_DATA_DELETED (replaced by tablet 799780bf376843499c659df4cd2b4aa1) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:23.276101 1483 tablet_service.cc:1505] Processing CreateTablet for tablet f5c3483130e84b66b0e5e4784701ffbf (DEFAULT_TABLE table=default.Table_3e12a4891d024f54bb915e9f4dd0c60f [id=4852ccffc0284e1d83557455f4f1e9f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.276522 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c3483130e84b66b0e5e4784701ffbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.282423 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b5f8df062a424c1f87d85918c22ebd13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.283083 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5f8df062a424c1f87d85918c22ebd13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.284163 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4846799df5b242ce8f19c034320c8a1d
I20251024 19:48:23.287122 1473 tablet_service.cc:1505] Processing CreateTablet for tablet e6fd158e8b0c4cc78cf7fd3b34d5f104 (DEFAULT_TABLE table=default.Table_67e6d3d8a4334247a1bf237d5936799b [id=b0c7cbd53bbd4b9fbaef10e6c8604cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.287487 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6fd158e8b0c4cc78cf7fd3b34d5f104. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:23.288733 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9595525b6e594761ae90d7fcde6eaea4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.290048 1626 tablet_service.cc:1505] Processing CreateTablet for tablet d27750aaf3c84f5eae2df5f4337063c3 (DEFAULT_TABLE table=default.Table_3b12fc2aa8564985980b459691c71555 [id=1ad6109902444b2ab657b1d7f0d86cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.290565 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27750aaf3c84f5eae2df5f4337063c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.290649 1189 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet fbda0935c4614aae9a57fc0f71497508
I20251024 19:48:23.293104 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 81f8393f78204860bd854f4872099387
I20251024 19:48:23.294368 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 3a74d7661a5f445393b27e47744ecdf6 (DEFAULT_TABLE table=default.Table_9bcef35dbbd649ccbf09b1067e78bcc8 [id=c57bfdb496874a67a90e601c972818d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.294895 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a74d7661a5f445393b27e47744ecdf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.295478 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ea0c3c7222394ffda2136fe114e6760b" } new_table_name: "default.Table_303b0da74b284bc5a810b08b52f820ba" modify_external_catalogs: true
I20251024 19:48:23.296394 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6cfafb2b827d4dba9de0de5bb5472304" } new_table_name: "default.Table_06fa0bf1c93d492d82dc0e412d9418b3" modify_external_catalogs: true
I20251024 19:48:23.296412 1767 tablet_bootstrap.cc:654] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:23.300899 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 0276f82a2e78458f9d065c94c6d32edc (DEFAULT_TABLE table=default.Table_8f9e49a43cb94787a02a10d3017b4e75 [id=1f841222d7264cfa849e8132aa3e5db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.301266 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0276f82a2e78458f9d065c94c6d32edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.301652 1766 tablet_bootstrap.cc:492] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:23.301795 1766 ts_tablet_manager.cc:1403] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.381s user 0.004s sys 0.000s
I20251024 19:48:23.302769 1766 raft_consensus.cc:359] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:23.302946 1766 raft_consensus.cc:385] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:23.303026 1766 raft_consensus.cc:740] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:23.303392 1766 consensus_queue.cc:260] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:23.304533 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96a6e0a4882c42399fd07f06ac21aa3f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:23.304639 1766 ts_tablet_manager.cc:1434] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251024 19:48:23.305601 1611 tablet_service.cc:1505] Processing CreateTablet for tablet f5c3483130e84b66b0e5e4784701ffbf (DEFAULT_TABLE table=default.Table_3e12a4891d024f54bb915e9f4dd0c60f [id=4852ccffc0284e1d83557455f4f1e9f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.306058 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c3483130e84b66b0e5e4784701ffbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.306373 1305 leader_election.cc:304] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:23.306846 3374 raft_consensus.cc:2749] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:23.309890 1477 tablet_service.cc:1505] Processing CreateTablet for tablet f5282515c1844ae4b7469551436586c2 (DEFAULT_TABLE table=default.Table_941306808730402d8ac4181a7a4877f2 [id=0bd0ebff5a934532a9a284caf4245eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.310238 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5282515c1844ae4b7469551436586c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.313105 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet b24a3a820c9b496dbf6955b399c6ea8c with delete_type TABLET_DATA_DELETED (replaced by tablet 799780bf376843499c659df4cd2b4aa1) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:23.314296 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 97f6223d9e8b4f80b70b7515108d8492 consensus state has no leader
I20251024 19:48:23.314450 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 42c6fbc3067d433489230906816fc202 (DEFAULT_TABLE table=default.Table_9db3b592a2a64917b67ad65189a48a88 [id=b4b24248eaf6415488598c46f1cd2b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.314793 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c6fbc3067d433489230906816fc202. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:23.315034 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be21ca442dee4204a8b15f5a2dd9530b consensus state has no leader
I20251024 19:48:23.319460 1621 tablet_service.cc:1505] Processing CreateTablet for tablet e6fd158e8b0c4cc78cf7fd3b34d5f104 (DEFAULT_TABLE table=default.Table_67e6d3d8a4334247a1bf237d5936799b [id=b0c7cbd53bbd4b9fbaef10e6c8604cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.320720 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6fd158e8b0c4cc78cf7fd3b34d5f104. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.325006 1775 ts_tablet_manager.cc:1929] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:23.325151 1775 log.cc:1199] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/f3d80885fcc642b8a75da30cd4abecbd
I20251024 19:48:23.326434 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eafde5a40df4fbfb03340b7bf791a2a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:23.328198 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_510bc2b3752b42b09ebef315e1e79a48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.328823 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_510bc2b3752b42b09ebef315e1e79a48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.329124 1364 raft_consensus.cc:3060] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:23.330077 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_200b6d5fd015449d877cd90dba49089f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.331043 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_200b6d5fd015449d877cd90dba49089f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.329427 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 19acf59898c8436e9d9fef98a7f07532 (DEFAULT_TABLE table=default.Table_dab0576c96374dd0b1cd1409c1594039 [id=f2f912a6c50e4fcdad32d3b096edbf97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.331740 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19acf59898c8436e9d9fef98a7f07532. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.331449 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9740a4661e9345ccbf754fbd167df7f6" } new_table_name: "default.Table_e63a801ea6b24349a2f613e71d13a561" modify_external_catalogs: true
I20251024 19:48:23.333215 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_761f1d6d97dc46c28951c41e9c5082b3" } modify_external_catalogs: true
I20251024 19:48:23.337742 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_761f1d6d97dc46c28951c41e9c5082b3" } modify_external_catalogs: true
I20251024 19:48:23.342296 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5b54ca8cd83f49188876e061f70d629e" } new_table_name: "default.Table_3b64df5b0ace46d59215a38eacb8c833" modify_external_catalogs: true
I20251024 19:48:23.344370 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 3a74d7661a5f445393b27e47744ecdf6 (DEFAULT_TABLE table=default.Table_9bcef35dbbd649ccbf09b1067e78bcc8 [id=c57bfdb496874a67a90e601c972818d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.345381 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 0276f82a2e78458f9d065c94c6d32edc (DEFAULT_TABLE table=default.Table_8f9e49a43cb94787a02a10d3017b4e75 [id=1f841222d7264cfa849e8132aa3e5db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.345966 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8488a13f677347b591b4efde0761451b" } new_table_name: "default.Table_6d733746a11f4a1eb01636fe7d283523" modify_external_catalogs: true
I20251024 19:48:23.346730 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a74d7661a5f445393b27e47744ecdf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.350301 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3b59a05b195a4d728ae60c0e25c5037f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.350909 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b59a05b195a4d728ae60c0e25c5037f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.333464 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 08de859a187b4f978d1925607d2c6402 (DEFAULT_TABLE table=default.Table_c80fa7789cc34b718c6de9760db48a17 [id=465a7cd9fcaf4a03910b7fad5abea2e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.355446 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08de859a187b4f978d1925607d2c6402. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.325577 1775 ts_tablet_manager.cc:1950] T f3d80885fcc642b8a75da30cd4abecbd P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:23.358035 1619 tablet_service.cc:1505] Processing CreateTablet for tablet c70bfd01ae774245b42810722e508b46 (DEFAULT_TABLE table=default.Table_d604f7e8552840359eae4a05ca6b65c5 [id=3f87372b020a44a89a6c9b7687f19c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.336472 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 830c8469331940fc85aaad58c9085e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:23.336999 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 4846799df5b242ce8f19c034320c8a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:23.340678 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 81f8393f78204860bd854f4872099387 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:23.359730 1630 tablet_service.cc:1505] Processing CreateTablet for tablet f5282515c1844ae4b7469551436586c2 (DEFAULT_TABLE table=default.Table_941306808730402d8ac4181a7a4877f2 [id=0bd0ebff5a934532a9a284caf4245eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.343839 1474 tablet_service.cc:1505] Processing CreateTablet for tablet e111980a15894b5084d9384297389b5d (DEFAULT_TABLE table=default.Table_5246796e5d1f4fc9b7e766d6bfa2ae29 [id=32c23c7e2157439e985f152cd803f1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.360236 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e111980a15894b5084d9384297389b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.360939 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 42c6fbc3067d433489230906816fc202 (DEFAULT_TABLE table=default.Table_9db3b592a2a64917b67ad65189a48a88 [id=b4b24248eaf6415488598c46f1cd2b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.361706 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65424b00ab9c454f9584048caba83e6a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:23.361719 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7aba73c5ba134980be24d93cae8e5f45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:23.362185 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 08de859a187b4f978d1925607d2c6402 (DEFAULT_TABLE table=default.Table_c80fa7789cc34b718c6de9760db48a17 [id=465a7cd9fcaf4a03910b7fad5abea2e3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:23.362396 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7aba73c5ba134980be24d93cae8e5f45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.363823 1627 tablet_service.cc:1505] Processing CreateTablet for tablet e111980a15894b5084d9384297389b5d (DEFAULT_TABLE table=default.Table_5246796e5d1f4fc9b7e766d6bfa2ae29 [id=32c23c7e2157439e985f152cd803f1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.358829 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 19acf59898c8436e9d9fef98a7f07532 (DEFAULT_TABLE table=default.Table_dab0576c96374dd0b1cd1409c1594039 [id=f2f912a6c50e4fcdad32d3b096edbf97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.344949 1506 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eafde5a40df4fbfb03340b7bf791a2a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:23.345973 2858 raft_consensus.cc:493] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:23.364275 2858 raft_consensus.cc:515] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:23.364903 2858 leader_election.cc:290] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:23.348824 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 675301c87d784b0a8ffe04404960be58 (DEFAULT_TABLE table=default.Table_8d0cf72d8a64479cb61a8f1317f392c1 [id=22aa0e3d850f46028f0e122be23922c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.365311 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 675301c87d784b0a8ffe04404960be58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.335450 1483 tablet_service.cc:1505] Processing CreateTablet for tablet c70bfd01ae774245b42810722e508b46 (DEFAULT_TABLE table=default.Table_d604f7e8552840359eae4a05ca6b65c5 [id=3f87372b020a44a89a6c9b7687f19c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.366003 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c70bfd01ae774245b42810722e508b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.370134 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc7ccf1495924782b9b6ac1173bc73a1" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:23.370626 1444 leader_election.cc:343] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:23.347608 1472 tablet_service.cc:1505] Processing CreateTablet for tablet 043fac06e6324032a61c42a10d51ea22 (DEFAULT_TABLE table=default.Table_98de64dffcd34afc91c487faa22cb705 [id=d9ae61ad6f8545d8a9945f08bdd731a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.370818 1776 ts_tablet_manager.cc:1929] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:23.370945 1776 log.cc:1199] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/ee68243b4a664575b679950603bb250e
I20251024 19:48:23.371397 1776 ts_tablet_manager.cc:1950] T ee68243b4a664575b679950603bb250e P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:23.372195 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1035) took 2918 ms. Trace:
I20251024 19:48:23.372287 1776 rpcz_store.cc:276] 1024 19:48:20.453797 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.453856 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:23.208875 (+2755019us) ts_tablet_manager.cc:1151] Deleting tablet ee68243b4a664575b679950603bb250e
1024 19:48:23.208886 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:23.282307 (+ 73421us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.347619 (+ 65312us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.372176 (+ 24557us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2754814}
W20251024 19:48:23.377508 1305 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 71190.7us
I20251024 19:48:23.378899 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2d8b8a1cff3540f7b7f13712425d534e
I20251024 19:48:23.380937 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19acf59898c8436e9d9fef98a7f07532. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.381629 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_55de88bea3af4f0f8cc0c40bf6dbc127"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.382386 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55de88bea3af4f0f8cc0c40bf6dbc127 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.383812 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1999b751e9344dcf81d26226c0589c8f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:23.384755 1776 tablet_replica.cc:333] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:48:23.380133 1447 leader_election.cc:343] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:23.385713 1447 leader_election.cc:304] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:23.382589 1503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65424b00ab9c454f9584048caba83e6a" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:23.386014 1503 raft_consensus.cc:3060] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:23.364142 1506 raft_consensus.cc:3060] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:23.387476 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ff0ec53ae71461baf2b8f9ec065530f consensus state has no leader
W20251024 19:48:23.388907 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e80dfddd21b41e98ebd226e0219b519 consensus state has no leader
I20251024 19:48:23.389174 1503 raft_consensus.cc:2468] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
W20251024 19:48:23.390218 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 583d0ef19aad4a80a8da15f7abc481c2 consensus state has no leader
I20251024 19:48:23.391316 2858 raft_consensus.cc:2749] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:23.392362 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 043fac06e6324032a61c42a10d51ea22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.387041 1776 raft_consensus.cc:2243] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:23.397861 1776 raft_consensus.cc:2272] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:23.398852 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_02cc94782c6f48e5ae8aef6e9e879583"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.399509 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02cc94782c6f48e5ae8aef6e9e879583 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.399993 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1999b751e9344dcf81d26226c0589c8f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:23.401903 1346 tablet_service.cc:1505] Processing CreateTablet for tablet c70bfd01ae774245b42810722e508b46 (DEFAULT_TABLE table=default.Table_d604f7e8552840359eae4a05ca6b65c5 [id=3f87372b020a44a89a6c9b7687f19c98]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:23.402545 1447 leader_election.cc:343] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:23.403815 1341 tablet_service.cc:1505] Processing CreateTablet for tablet f5282515c1844ae4b7469551436586c2 (DEFAULT_TABLE table=default.Table_941306808730402d8ac4181a7a4877f2 [id=0bd0ebff5a934532a9a284caf4245eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.410526 1776 ts_tablet_manager.cc:1916] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:23.420626 1767 tablet_bootstrap.cc:492] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:23.428818 1447 leader_election.cc:304] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:23.429744 2858 raft_consensus.cc:2749] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:23.400756 1505 raft_consensus.cc:2468] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:23.421368 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1012) took 3368 ms. Trace:
I20251024 19:48:23.436884 1775 rpcz_store.cc:276] 1024 19:48:20.052660 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.052732 (+ 72us) service_pool.cc:225] Handling call
1024 19:48:22.806233 (+2753501us) ts_tablet_manager.cc:1151] Deleting tablet f3d80885fcc642b8a75da30cd4abecbd
1024 19:48:22.806248 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:23.063159 (+256911us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.324872 (+261713us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.421334 (+ 96462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2753288}
I20251024 19:48:23.437548 1472 tablet_service.cc:1505] Processing CreateTablet for tablet 04dfa03ab59e4891ad5edd4003b9ceb8 (DEFAULT_TABLE table=default.Table_423e7aaec4b54cb79a21504a67d8b143 [id=3c7b3acb377e4fe7838248958fd5c398]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.438388 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04dfa03ab59e4891ad5edd4003b9ceb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.444336 1766 tablet_bootstrap.cc:492] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:23.446506 3374 raft_consensus.cc:493] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:23.446660 3374 raft_consensus.cc:515] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.447281 3374 leader_election.cc:290] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:23.450830 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1fbbba88a294adfbf97c0343d86698e" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:23.451619 1505 raft_consensus.cc:2468] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:23.453014 1308 leader_election.cc:304] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:23.453401 3374 raft_consensus.cc:2804] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251024 19:48:23.453509 3374 raft_consensus.cc:493] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:23.453711 3374 raft_consensus.cc:3060] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Advancing to term 2
I20251024 19:48:23.456444 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 42c6fbc3067d433489230906816fc202 (DEFAULT_TABLE table=default.Table_9db3b592a2a64917b67ad65189a48a88 [id=b4b24248eaf6415488598c46f1cd2b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.456969 3374 raft_consensus.cc:515] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.458779 1333 tablet_service.cc:1505] Processing CreateTablet for tablet 08de859a187b4f978d1925607d2c6402 (DEFAULT_TABLE table=default.Table_c80fa7789cc34b718c6de9760db48a17 [id=465a7cd9fcaf4a03910b7fad5abea2e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.458896 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1fbbba88a294adfbf97c0343d86698e" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:23.459050 1507 raft_consensus.cc:3060] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Advancing to term 2
I20251024 19:48:23.459679 1350 tablet_service.cc:1505] Processing CreateTablet for tablet e111980a15894b5084d9384297389b5d (DEFAULT_TABLE table=default.Table_5246796e5d1f4fc9b7e766d6bfa2ae29 [id=32c23c7e2157439e985f152cd803f1af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.460461 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 0276f82a2e78458f9d065c94c6d32edc (DEFAULT_TABLE table=default.Table_8f9e49a43cb94787a02a10d3017b4e75 [id=1f841222d7264cfa849e8132aa3e5db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.462324 1507 raft_consensus.cc:2468] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 2.
I20251024 19:48:23.462793 1308 leader_election.cc:304] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:23.463057 3296 raft_consensus.cc:2804] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [term 2 FOLLOWER]: Leader election won for term 2
I20251024 19:48:23.464556 1767 ts_tablet_manager.cc:1403] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.504s user 0.000s sys 0.007s
I20251024 19:48:23.465986 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 830c8469331940fc85aaad58c9085e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:23.466711 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 4846799df5b242ce8f19c034320c8a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:23.467324 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 1edc55caab904722bf19a1c71df601cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:23.468071 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 81f8393f78204860bd854f4872099387 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:23.472000 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1312dfdab750454a8f412ce6a66284ac" } modify_external_catalogs: true
I20251024 19:48:23.472538 3374 leader_election.cc:290] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 2 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:23.472674 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1312dfdab750454a8f412ce6a66284ac" } modify_external_catalogs: true
I20251024 19:48:23.474051 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_84dc488a8da34da79f25c6ac2c36beca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.475386 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84dc488a8da34da79f25c6ac2c36beca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.482263 2923 consensus_queue.cc:1048] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:23.484591 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 043fac06e6324032a61c42a10d51ea22 (DEFAULT_TABLE table=default.Table_98de64dffcd34afc91c487faa22cb705 [id=d9ae61ad6f8545d8a9945f08bdd731a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.486161 1348 tablet_service.cc:1505] Processing CreateTablet for tablet 675301c87d784b0a8ffe04404960be58 (DEFAULT_TABLE table=default.Table_8d0cf72d8a64479cb61a8f1317f392c1 [id=22aa0e3d850f46028f0e122be23922c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.486270 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_34b29b41568a41e8a7cd22e7ad65949f" } modify_external_catalogs: true
I20251024 19:48:23.486449 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d8b8a1cff3540f7b7f13712425d534e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:23.486547 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_34b29b41568a41e8a7cd22e7ad65949f" } modify_external_catalogs: true
I20251024 19:48:23.487689 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 04dfa03ab59e4891ad5edd4003b9ceb8 (DEFAULT_TABLE table=default.Table_423e7aaec4b54cb79a21504a67d8b143 [id=3c7b3acb377e4fe7838248958fd5c398]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.493983 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 81f8393f78204860bd854f4872099387 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:23.494655 1191 master_service.cc:954] ReplaceTablet: received request to replace tablet 9229fa6bf13441fa894e7bb7f78d7038 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:23.495577 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_effb50443d274effaa5039639c921715" } modify_external_catalogs: true
I20251024 19:48:23.495790 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_effb50443d274effaa5039639c921715" } modify_external_catalogs: true
I20251024 19:48:23.496804 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_07d8ae994ade4899b0da421c94480b75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.497510 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07d8ae994ade4899b0da421c94480b75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.437053 1483 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d8b8a1cff3540f7b7f13712425d534e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:23.437747 1775 tablet_replica.cc:333] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:23.506726 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet 830c8469331940fc85aaad58c9085e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:23.508347 1506 raft_consensus.cc:2468] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:23.509387 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 043fac06e6324032a61c42a10d51ea22 (DEFAULT_TABLE table=default.Table_98de64dffcd34afc91c487faa22cb705 [id=d9ae61ad6f8545d8a9945f08bdd731a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.511287 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 3440f176bd814230b220b32586cc1208 (DEFAULT_TABLE table=default.Table_b952a3ced9b94e19bcee920f864b326c [id=457fabac72bc45d6bdb0c124e42cedaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.511710 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 505ba3165385434faea556e97f42a7f6 (DEFAULT_TABLE table=default.Table_bc3b64103c5b40db85bb106ea58c5c62 [id=9153ba48cc8e468589aae7aa0ba899ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.510660 1628 tablet_service.cc:1505] Processing CreateTablet for tablet b4b7defa4cac4f62988eb0a0b1227997 (DEFAULT_TABLE table=default.Table_f9f3cd8a1e5748f6a9d6b77180996272 [id=831b5fb847ab42a8b60d830226f599ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.510869 1629 tablet_service.cc:1505] Processing CreateTablet for tablet d9ee0844646e4034b45f307738623968 (DEFAULT_TABLE table=default.Table_e845d12029a948f7b361c33a3cdabfc2 [id=fa665c45a694480f8b4bb2aeea2d08af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.512253 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9026d9e801294dd2bfdb9c90e45a0990" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:23.510442 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 04dfa03ab59e4891ad5edd4003b9ceb8 (DEFAULT_TABLE table=default.Table_423e7aaec4b54cb79a21504a67d8b143 [id=3c7b3acb377e4fe7838248958fd5c398]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.512490 1642 raft_consensus.cc:2468] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:23.511085 1620 tablet_service.cc:1505] Processing CreateTablet for tablet f9379d71d772424891ed8d3d72cafd6d (DEFAULT_TABLE table=default.Table_f65ea29d17fe40599ad9377f12464b7f [id=eb2d43c20f9747b3b223349cc42963e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.518291 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet ee68243b4a664575b679950603bb250e (table default.Table_605117e2eb734e6b8dc34ceb4b5b5b71 [id=38b6e1273e1a43bd8b12604db5371677]) successfully deleted
I20251024 19:48:23.520886 1483 tablet_service.cc:1505] Processing CreateTablet for tablet b4b7defa4cac4f62988eb0a0b1227997 (DEFAULT_TABLE table=default.Table_f9f3cd8a1e5748f6a9d6b77180996272 [id=831b5fb847ab42a8b60d830226f599ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.521333 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b7defa4cac4f62988eb0a0b1227997. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.529361 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a5046911ec4642bd811676fbd8782d9a" } new_table_name: "default.Table_dbc6b2ebea18461698c8f3bcd4d989bc" modify_external_catalogs: true
I20251024 19:48:23.531131 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2ead137126594dc39864dfac900057d0" } new_table_name: "default.Table_fc6e97f574374fe4972a47db87f70001" modify_external_catalogs: true
I20251024 19:48:23.532583 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_32612c9da3dd4bce8d35d2fdc3bb4bd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.533313 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32612c9da3dd4bce8d35d2fdc3bb4bd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.534678 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_02497c364ee24472a2d1ca697b81e22e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.535285 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02497c364ee24472a2d1ca697b81e22e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.540503 1349 tablet_service.cc:1505] Processing CreateTablet for tablet b4b7defa4cac4f62988eb0a0b1227997 (DEFAULT_TABLE table=default.Table_f9f3cd8a1e5748f6a9d6b77180996272 [id=831b5fb847ab42a8b60d830226f599ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.548400 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9e22d18b516543cf9c179d97a1665544" } modify_external_catalogs: true
W20251024 19:48:23.554711 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f2df2b286024ae8be02b29041c36803 has no consensus state
W20251024 19:48:23.561815 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ec3afdbd80848e19fb0c4cd5b0eb0e7 consensus state has no leader
I20251024 19:48:23.561980 1490 tablet_service.cc:1505] Processing CreateTablet for tablet d9ee0844646e4034b45f307738623968 (DEFAULT_TABLE table=default.Table_e845d12029a948f7b361c33a3cdabfc2 [id=fa665c45a694480f8b4bb2aeea2d08af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.562454 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ee0844646e4034b45f307738623968. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.572034 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 3440f176bd814230b220b32586cc1208 (DEFAULT_TABLE table=default.Table_b952a3ced9b94e19bcee920f864b326c [id=457fabac72bc45d6bdb0c124e42cedaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.572345 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 505ba3165385434faea556e97f42a7f6 (DEFAULT_TABLE table=default.Table_bc3b64103c5b40db85bb106ea58c5c62 [id=9153ba48cc8e468589aae7aa0ba899ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.572475 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3440f176bd814230b220b32586cc1208. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.572713 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505ba3165385434faea556e97f42a7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.572036 1485 tablet_service.cc:1505] Processing CreateTablet for tablet f9379d71d772424891ed8d3d72cafd6d (DEFAULT_TABLE table=default.Table_f65ea29d17fe40599ad9377f12464b7f [id=eb2d43c20f9747b3b223349cc42963e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.573064 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9379d71d772424891ed8d3d72cafd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.578274 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 675301c87d784b0a8ffe04404960be58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.578990 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_097afacca75e46dfbfec9dd862b21a6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:23.579201 1768 tablet_bootstrap.cc:654] T 385e71e672154e399f14321eac0d8625 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:23.581739 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06212ab82546412d9412415159932179. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.582569 1618 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d8b8a1cff3540f7b7f13712425d534e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:23.583364 1333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08de859a187b4f978d1925607d2c6402. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.585685 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04dfa03ab59e4891ad5edd4003b9ceb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.587455 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4f639b4e1ca94db3a70d5ce8a2d5b050"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:23.588517 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2691f5c07d014467892d86561d71800b" } modify_external_catalogs: true
I20251024 19:48:23.588881 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2691f5c07d014467892d86561d71800b" } modify_external_catalogs: true
I20251024 19:48:23.589702 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 043fac06e6324032a61c42a10d51ea22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.589844 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6292d44b4a4b4458b784334afc4ed28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.593740 1341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5282515c1844ae4b7469551436586c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.594707 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_35b260ec41dd461eb1b08fd8485c69c9" } new_table_name: "default.Table_a65cc71737804784befd07c58aefdd0b" modify_external_catalogs: true
I20251024 19:48:23.596429 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_34c8fbbe53954b97b7ead671ade3783b" } new_table_name: "default.Table_a6e229dd5966441ab60d62ca8e0dfe16" modify_external_catalogs: true
I20251024 19:48:23.596571 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b7defa4cac4f62988eb0a0b1227997. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.597332 1774 ts_tablet_manager.cc:1929] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:23.597934 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c6fbc3067d433489230906816fc202. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.598426 1774 log.cc:1199] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/d2a68bb1dc6c483388784b9bb19ce065
I20251024 19:48:23.600474 1774 ts_tablet_manager.cc:1950] T d2a68bb1dc6c483388784b9bb19ce065 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:23.601485 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e111980a15894b5084d9384297389b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.550576 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9e22d18b516543cf9c179d97a1665544" } modify_external_catalogs: true
W20251024 19:48:23.606104 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f639b4e1ca94db3a70d5ce8a2d5b050 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.607553 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a74d7661a5f445393b27e47744ecdf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.608305 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 675301c87d784b0a8ffe04404960be58 (DEFAULT_TABLE table=default.Table_8d0cf72d8a64479cb61a8f1317f392c1 [id=22aa0e3d850f46028f0e122be23922c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.608951 1775 raft_consensus.cc:2243] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:23.609375 1775 raft_consensus.cc:2272] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:23.610337 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a98058c346f4bb89d07f7409052aade consensus state has no leader
I20251024 19:48:23.610778 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet f3d80885fcc642b8a75da30cd4abecbd (table default.Table_7966ff9879d24254aed39562b2f127d7 [id=45b38d5e9fd6472fa318b41863080708]) successfully deleted
I20251024 19:48:23.610827 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4bd52fc39eef4537bafea34b959fa79c" } new_table_name: "default.Table_8e1e1975e9ec45ddadd9bdd0f04026bb" modify_external_catalogs: true
I20251024 19:48:23.612190 1340 tablet_service.cc:1505] Processing CreateTablet for tablet d9ee0844646e4034b45f307738623968 (DEFAULT_TABLE table=default.Table_e845d12029a948f7b361c33a3cdabfc2 [id=fa665c45a694480f8b4bb2aeea2d08af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.612872 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet 4846799df5b242ce8f19c034320c8a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:23.613687 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6fd158e8b0c4cc78cf7fd3b34d5f104. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:23.614673 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe8656f221984776ad3978b9e651654c has no consensus state
I20251024 19:48:23.617690 1335 tablet_service.cc:1505] Processing CreateTablet for tablet f9379d71d772424891ed8d3d72cafd6d (DEFAULT_TABLE table=default.Table_f65ea29d17fe40599ad9377f12464b7f [id=eb2d43c20f9747b3b223349cc42963e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.617986 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_97c3d690de704e908a7ca2e664236bd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.619652 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97c3d690de704e908a7ca2e664236bd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.620779 3296 raft_consensus.cc:697] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [term 2 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:23.621161 3296 consensus_queue.cc:237] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.622804 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19acf59898c8436e9d9fef98a7f07532. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.623296 1775 ts_tablet_manager.cc:1916] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:23.623773 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 3440f176bd814230b220b32586cc1208 (DEFAULT_TABLE table=default.Table_b952a3ced9b94e19bcee920f864b326c [id=457fabac72bc45d6bdb0c124e42cedaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:23.624861 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 505ba3165385434faea556e97f42a7f6 (DEFAULT_TABLE table=default.Table_bc3b64103c5b40db85bb106ea58c5c62 [id=9153ba48cc8e468589aae7aa0ba899ec]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:23.627027 1447 leader_election.cc:343] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:23.627215 1447 leader_election.cc:304] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:23.627573 3390 raft_consensus.cc:2749] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:23.634892 1766 tablet_bootstrap.cc:654] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:23.637723 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c70bfd01ae774245b42810722e508b46. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:23.639110 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06212ab82546412d9412415159932179 consensus state has no leader
I20251024 19:48:23.639744 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_40837137d2a145828e6dcbd42e88e393"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.644001 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40837137d2a145828e6dcbd42e88e393 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.649770 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0276f82a2e78458f9d065c94c6d32edc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:23.650928 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_097afacca75e46dfbfec9dd862b21a6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.653084 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1fbbba88a294adfbf97c0343d86698e" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:23.653651 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1fbbba88a294adfbf97c0343d86698e" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
W20251024 19:48:23.510030 1586 leader_election.cc:343] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:23.656952 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 173504us
W20251024 19:48:23.657166 1586 leader_election.cc:343] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:23.657456 1586 leader_election.cc:343] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:23.657799 1586 leader_election.cc:343] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:23.657991 1586 leader_election.cc:343] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:23.662323 1766 tablet_bootstrap.cc:492] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:23.664067 1766 ts_tablet_manager.cc:1403] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.220s user 0.005s sys 0.004s
I20251024 19:48:23.669461 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 999) took 1151 ms. Trace:
I20251024 19:48:23.669600 1774 rpcz_store.cc:276] 1024 19:48:22.517695 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.517751 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:23.111852 (+594101us) ts_tablet_manager.cc:1151] Deleting tablet d2a68bb1dc6c483388784b9bb19ce065
1024 19:48:23.111862 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:23.583687 (+471825us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.596925 (+ 13238us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.669435 (+ 72510us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":593942}
I20251024 19:48:23.679597 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc7ccf1495924782b9b6ac1173bc73a1" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:23.688477 1766 raft_consensus.cc:359] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:23.690948 1766 raft_consensus.cc:385] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:23.691574 1766 raft_consensus.cc:740] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:23.693051 1766 consensus_queue.cc:260] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:23.697899 1766 ts_tablet_manager.cc:1434] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.033s user 0.000s sys 0.004s
I20251024 19:48:23.700469 1766 tablet_bootstrap.cc:492] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:23.702148 1774 tablet_replica.cc:333] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:23.702349 1774 raft_consensus.cc:2243] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:23.702528 1774 raft_consensus.cc:2272] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:23.703601 1774 ts_tablet_manager.cc:1916] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:23.708652 1767 raft_consensus.cc:359] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.709064 1767 raft_consensus.cc:385] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:23.709317 1767 raft_consensus.cc:740] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:23.704941 1305 leader_election.cc:304] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
I20251024 19:48:23.710601 1767 consensus_queue.cc:260] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.721227 3296 raft_consensus.cc:2804] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:23.721357 3296 raft_consensus.cc:493] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:23.721421 3296 raft_consensus.cc:3060] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:23.723495 1187 catalog_manager.cc:5649] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 reported cstate change: term changed from 1 to 2, leader changed from af3952de050140d383f6b0de7666fa97 (127.0.252.195) to 43bec2f03fc849099e254f6335e9d532 (127.0.252.193). New cstate: current_term: 2 leader_uuid: "43bec2f03fc849099e254f6335e9d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:23.724037 3296 raft_consensus.cc:515] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.724612 3296 leader_election.cc:290] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:23.724807 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet fc7ccf1495924782b9b6ac1173bc73a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:23.723675 1766 tablet_bootstrap.cc:654] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:23.729395 1308 leader_election.cc:343] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 9026d9e801294dd2bfdb9c90e45a0990
I20251024 19:48:23.740074 1775 ts_tablet_manager.cc:1929] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:23.746737 1768 tablet_bootstrap.cc:492] T 385e71e672154e399f14321eac0d8625 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:23.747227 1768 ts_tablet_manager.cc:1403] T 385e71e672154e399f14321eac0d8625 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.488s user 0.028s sys 0.025s
I20251024 19:48:23.751176 1768 raft_consensus.cc:359] T 385e71e672154e399f14321eac0d8625 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:23.753307 1768 raft_consensus.cc:385] T 385e71e672154e399f14321eac0d8625 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:23.754791 1768 raft_consensus.cc:740] T 385e71e672154e399f14321eac0d8625 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:23.761227 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ee0844646e4034b45f307738623968. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.764446 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3440f176bd814230b220b32586cc1208. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.771561 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505ba3165385434faea556e97f42a7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.715051 1767 ts_tablet_manager.cc:1434] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.250s user 0.004s sys 0.001s
I20251024 19:48:23.777772 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9379d71d772424891ed8d3d72cafd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:23.748960 3360 consensus_queue.cc:1048] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20251024 19:48:23.756771 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9026d9e801294dd2bfdb9c90e45a0990" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
W20251024 19:48:23.748960 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 90298.7us
I20251024 19:48:23.778307 1642 raft_consensus.cc:3060] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:23.778508 1586 leader_election.cc:304] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:23.778995 3387 raft_consensus.cc:2804] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:23.779054 1586 leader_election.cc:304] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:23.779379 3141 raft_consensus.cc:2804] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:23.779476 3141 raft_consensus.cc:697] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:23.779709 3141 consensus_queue.cc:237] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.779068 3387 raft_consensus.cc:493] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:23.780727 3387 raft_consensus.cc:3060] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:23.781241 1642 raft_consensus.cc:2468] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:23.783584 3387 raft_consensus.cc:515] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.784212 3387 leader_election.cc:290] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:23.785935 1767 tablet_bootstrap.cc:492] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:23.796883 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_46ccc3bbcdad400884c16fd5425e5c3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.797331 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 8ff0ec53ae71461baf2b8f9ec065530f (table default.Table_e63a801ea6b24349a2f613e71d13a561 [id=64f9a8f7a56c4a32a12059f106d879c1], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8ff0ec53ae71461baf2b8f9ec065530f consensus state has no leader
I20251024 19:48:23.800084 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5be196925fd1419998ee4e85f267f8a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:23.749150 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet d2a68bb1dc6c483388784b9bb19ce065 (table default.Table_3be1c2c5954947969e6903a3f016d589 [id=0b304e4408b24fe797f4438b874e480f]) successfully deleted
W20251024 19:48:23.800482 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 51376.1us
W20251024 19:48:23.800626 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5be196925fd1419998ee4e85f267f8a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.741115 1775 log.cc:1199] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/28005301f9de48e5826f46c79aeb5fe5
W20251024 19:48:23.749333 1447 leader_election.cc:343] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:23.803654 1447 leader_election.cc:304] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:23.803974 1447 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 54655.2us
I20251024 19:48:23.804095 2858 raft_consensus.cc:2749] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:23.804319 1775 ts_tablet_manager.cc:1950] T 28005301f9de48e5826f46c79aeb5fe5 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
W20251024 19:48:23.804582 1447 consensus_peers.cc:597] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edb49884389c4dba957f0ee5af4ae0e4. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:23.804811 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 769f576810ab49368eb0fc7837b7da58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:23.806059 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_06e859c14ac948dd8996f7fbec456138"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.808390 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06e859c14ac948dd8996f7fbec456138 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:23.809664 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46ccc3bbcdad400884c16fd5425e5c3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.814057 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3db01f386bf3464cbfee13a63abf924a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.817852 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3db01f386bf3464cbfee13a63abf924a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.818359 1774 ts_tablet_manager.cc:1929] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:23.818598 1774 log.cc:1199] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/d840888183ec4ef7bb245363c5bac322
I20251024 19:48:23.819416 1774 ts_tablet_manager.cc:1950] T d840888183ec4ef7bb245363c5bac322 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:23.760599 1768 consensus_queue.cc:260] T 385e71e672154e399f14321eac0d8625 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:23.822984 1305 leader_election.cc:304] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
I20251024 19:48:23.823359 3296 raft_consensus.cc:2804] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:23.823637 3296 raft_consensus.cc:697] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:23.823992 3296 consensus_queue.cc:237] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.830626 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_34f366e7d1404f2e86acf32e02ae92ab" } new_table_name: "default.Table_13c5b3b8996b417bb8958cbc0e431b13" modify_external_catalogs: true
I20251024 19:48:23.837455 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1013) took 3784 ms. Trace:
I20251024 19:48:23.837977 1775 rpcz_store.cc:276] 1024 19:48:20.052948 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.052989 (+ 41us) service_pool.cc:225] Handling call
1024 19:48:23.437669 (+3384680us) ts_tablet_manager.cc:1151] Deleting tablet 28005301f9de48e5826f46c79aeb5fe5
1024 19:48:23.437695 (+ 26us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:23.685301 (+247606us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.738845 (+ 53544us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.837388 (+ 98543us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72709,"tablet-delete.queue_time_us":3384511}
I20251024 19:48:23.838963 1775 tablet_replica.cc:333] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:23.839380 1775 raft_consensus.cc:2243] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:23.839728 1775 pending_rounds.cc:70] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e: Trying to abort 1 pending ops.
I20251024 19:48:23.839871 1775 pending_rounds.cc:77] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429395179409408 op_type: NO_OP noop_request { }
I20251024 19:48:23.840092 1775 raft_consensus.cc:2889] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:23.840308 1775 raft_consensus.cc:2272] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:23.842880 1768 ts_tablet_manager.cc:1434] T 385e71e672154e399f14321eac0d8625 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.095s user 0.004s sys 0.000s
I20251024 19:48:23.843550 1768 tablet_bootstrap.cc:492] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:23.844803 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f8fd3d3a919d4e91844c5ded89bced1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.852173 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc50e9fbbefb43fda123ace9d1f32cf8 consensus state has no leader
I20251024 19:48:23.852803 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5fa0890a968b4ba785c99c435ffded4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:23.854189 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b668ef91cf2f47e7864a1fa580cda6d6" } new_table_name: "default.Table_89402da0de5c4575b677a5955a538306" modify_external_catalogs: true
I20251024 19:48:23.855221 1766 tablet_bootstrap.cc:492] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:23.855365 1766 ts_tablet_manager.cc:1403] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.156s user 0.006s sys 0.005s
I20251024 19:48:23.856068 1766 raft_consensus.cc:359] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:23.856226 1766 raft_consensus.cc:385] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:23.856276 1766 raft_consensus.cc:740] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:23.856494 1766 consensus_queue.cc:260] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:23.857502 1766 ts_tablet_manager.cc:1434] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251024 19:48:23.862324 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 28005301f9de48e5826f46c79aeb5fe5 (table default.Table_e74966425ce8408da939910bc645012d [id=e345af66b89e48f9afe96000ce50179c]) successfully deleted
I20251024 19:48:23.864391 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1000) took 1318 ms. Trace:
I20251024 19:48:23.864598 1774 rpcz_store.cc:276] 1024 19:48:22.545845 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.545949 (+ 104us) service_pool.cc:225] Handling call
1024 19:48:23.669933 (+1123984us) ts_tablet_manager.cc:1151] Deleting tablet d840888183ec4ef7bb245363c5bac322
1024 19:48:23.702093 (+ 32160us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:23.766204 (+ 64111us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.818087 (+ 51883us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.864336 (+ 46249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123731}
I20251024 19:48:23.865283 1774 tablet_replica.cc:333] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:23.865702 1774 raft_consensus.cc:2243] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:23.866040 1774 raft_consensus.cc:2272] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:23.869151 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8fd3d3a919d4e91844c5ded89bced1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:23.879015 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 018d3e6aba6740f0bd82e8c83863bc4e consensus state has no leader
I20251024 19:48:23.879454 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e3d3cfc2bb044e84acb9624615065375" } new_table_name: "default.Table_dc0f70b1b6cd4a62b93398c8b38ef1fe" modify_external_catalogs: true
I20251024 19:48:23.882501 1775 ts_tablet_manager.cc:1916] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:23.765281 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet a1fbbba88a294adfbf97c0343d86698e (table default.Table_7bc8cd4a930b44dfafa473297875594d [id=a60c8222aa0248c091fab8dc34100847]). Expected version 2 got 0
I20251024 19:48:23.887584 1766 tablet_bootstrap.cc:492] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:23.889030 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fa0890a968b4ba785c99c435ffded4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:23.889330 1766 tablet_bootstrap.cc:654] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:23.898921 1774 ts_tablet_manager.cc:1916] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:23.910488 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 03d8ab62b47f474198841ea14a3b7b00
I20251024 19:48:23.914315 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 03d8ab62b47f474198841ea14a3b7b00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:23.914753 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet 03d8ab62b47f474198841ea14a3b7b00 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:23.915272 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_84dc488a8da34da79f25c6ac2c36beca" } new_table_name: "default.Table_1fcb39e601194b41bfaaa87ec866ae47" modify_external_catalogs: true
I20251024 19:48:23.918397 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_915650e3f7984a0d81ae2f98d2b1d0a6" } new_table_name: "default.Table_5046f8911611433295fd660877c0c7e3" modify_external_catalogs: true
W20251024 19:48:23.924337 1011 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskc9NInd/build/asan/bin/kudu pid 1087: Timed out: CheckLeaks RPC to 127.0.252.253:41375 timed out after 1.000s (SENT)
I20251024 19:48:23.924642 1011 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskc9NInd/build/asan/bin/kudu with pid 1087
I20251024 19:48:23.925702 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2b21687caa4b4a0595f99684d3d95b22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:23.926335 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b21687caa4b4a0595f99684d3d95b22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:23.931258 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 151168us
W20251024 19:48:23.933063 1586 consensus_peers.cc:597] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d840888183ec4ef7bb245363c5bac322. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:23.935111 1586 leader_election.cc:304] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:23.937435 3141 raft_consensus.cc:2804] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:23.937661 3141 raft_consensus.cc:697] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:23.938069 3141 consensus_queue.cc:237] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.943802 1586 leader_election.cc:304] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:23.946796 3141 raft_consensus.cc:2804] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:23.947650 3141 raft_consensus.cc:493] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:23.951871 1768 tablet_bootstrap.cc:654] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:23.949030 3141 raft_consensus.cc:3060] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:23.968470 1586 leader_election.cc:304] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:23.972240 3404 raft_consensus.cc:2804] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:23.972524 3404 raft_consensus.cc:697] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:23.972765 3404 consensus_queue.cc:237] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.905892 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet d840888183ec4ef7bb245363c5bac322 (table default.Table_cb99a44a90e849be948f4a481c07a91d [id=5edc315e5e43438f83c57c449df681a8]) successfully deleted
W20251024 19:48:23.978598 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 73183.4us
I20251024 19:48:23.979314 3141 raft_consensus.cc:515] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.982255 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 03d8ab62b47f474198841ea14a3b7b00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:23.983443 1472 tablet_service.cc:1552] Processing DeleteTablet for tablet 03d8ab62b47f474198841ea14a3b7b00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:23.983626 1768 tablet_bootstrap.cc:492] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:23.984011 1768 ts_tablet_manager.cc:1403] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.141s user 0.006s sys 0.002s
I20251024 19:48:23.987339 1367 raft_consensus.cc:1240] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [term 2 LEADER]: Rejecting Update request from peer af3952de050140d383f6b0de7666fa97 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20251024 19:48:23.988221 1509 raft_consensus.cc:1240] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Rejecting Update request from peer af3952de050140d383f6b0de7666fa97 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20251024 19:48:23.988263 1369 raft_consensus.cc:3060] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:23.989008 3360 raft_consensus.cc:493] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:23.989161 3360 raft_consensus.cc:515] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.989522 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f965111ac60841dbb9e6ecc21ff3fec0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:23.989702 1510 raft_consensus.cc:3060] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:23.989804 3360 leader_election.cc:290] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:23.992414 1510 raft_consensus.cc:2468] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:23.994065 1366 raft_consensus.cc:3060] T 385e71e672154e399f14321eac0d8625 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:23.994382 1365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "250f106664be4b5aaa330dbb4e43cb7b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:23.994720 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "250f106664be4b5aaa330dbb4e43cb7b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:23.994863 1507 raft_consensus.cc:2468] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:23.996615 1766 tablet_bootstrap.cc:492] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:23.996743 1766 ts_tablet_manager.cc:1403] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.109s user 0.006s sys 0.002s
I20251024 19:48:23.997156 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f965111ac60841dbb9e6ecc21ff3fec0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:23.997474 1766 raft_consensus.cc:359] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.997675 1766 raft_consensus.cc:385] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:23.997733 1766 raft_consensus.cc:740] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:23.997423 1768 raft_consensus.cc:359] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.998131 1766 consensus_queue.cc:260] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:23.999280 1766 ts_tablet_manager.cc:1434] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:24.000186 1775 ts_tablet_manager.cc:1929] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:24.000509 1775 log.cc:1199] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/1edc55caab904722bf19a1c71df601cf
I20251024 19:48:24.001492 1775 ts_tablet_manager.cc:1950] T 1edc55caab904722bf19a1c71df601cf P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:23.998881 1768 raft_consensus.cc:385] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.005999 2923 raft_consensus.cc:493] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.004220 1768 raft_consensus.cc:740] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.006170 2923 raft_consensus.cc:515] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.006783 1766 tablet_bootstrap.cc:492] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:24.006879 2923 leader_election.cc:290] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:24.014312 1586 consensus_peers.cc:597] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2a68bb1dc6c483388784b9bb19ce065. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:24.010320 1768 consensus_queue.cc:260] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.014994 2923 consensus_queue.cc:1048] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:24.015173 3387 consensus_queue.cc:1059] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:24.015439 2923 raft_consensus.cc:3055] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Stepping down as leader of term 1
I20251024 19:48:24.015520 2923 raft_consensus.cc:740] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:24.015803 2923 consensus_queue.cc:260] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.020198 3141 leader_election.cc:290] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:24.024591 2923 raft_consensus.cc:3060] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Advancing to term 2
I20251024 19:48:24.026274 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 0a5034103c1e4e039df7fc9f0ab65d0c
I20251024 19:48:24.027303 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a5034103c1e4e039df7fc9f0ab65d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.027097 1768 ts_tablet_manager.cc:1434] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.043s user 0.005s sys 0.000s
W20251024 19:48:24.029116 1586 consensus_peers.cc:597] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:24.030115 1185 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (table default.Table_18e7285b7df44399ab3be1dfc86eb91f [id=14bda7f9324f44aa9cb87a5f129039be]). Expected version 2 got 0
I20251024 19:48:24.030086 1586 leader_election.cc:304] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:24.030696 2923 raft_consensus.cc:2804] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:24.030949 2923 raft_consensus.cc:697] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:24.031244 2923 consensus_queue.cc:237] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.034329 1768 tablet_bootstrap.cc:492] T f965111ac60841dbb9e6ecc21ff3fec0 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:24.035974 1586 consensus_peers.cc:597] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28005301f9de48e5826f46c79aeb5fe5. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:24.038369 1472 tablet_service.cc:1552] Processing DeleteTablet for tablet 80c58e2439dc490c824b5f0023b5434b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:24.038306 1586 leader_election.cc:304] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:24.039592 3387 raft_consensus.cc:2804] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:24.039680 3387 raft_consensus.cc:493] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:24.039743 3387 raft_consensus.cc:3060] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:24.042775 3387 raft_consensus.cc:515] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.043386 3387 leader_election.cc:290] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:24.044646 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1014) took 3991 ms. Trace:
I20251024 19:48:24.044799 1775 rpcz_store.cc:276] 1024 19:48:20.053228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.053270 (+ 42us) service_pool.cc:225] Handling call
1024 19:48:23.838874 (+3785604us) ts_tablet_manager.cc:1151] Deleting tablet 1edc55caab904722bf19a1c71df601cf
1024 19:48:23.838885 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:23.932739 (+ 93854us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:23.999921 (+ 67182us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.044622 (+ 44701us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3785176}
I20251024 19:48:24.045291 1775 tablet_replica.cc:333] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:24.045593 1775 raft_consensus.cc:2243] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:24.045805 1775 raft_consensus.cc:2272] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:24.047380 1775 ts_tablet_manager.cc:1916] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:24.045521 1766 tablet_bootstrap.cc:654] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:24.047962 1586 consensus_peers.cc:597] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3d80885fcc642b8a75da30cd4abecbd. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:24.053421 1186 catalog_manager.cc:5649] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:24.056002 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38818c7997c84cedad5f9b09eef77105" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:24.059063 1506 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38818c7997c84cedad5f9b09eef77105" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:24.059315 1506 raft_consensus.cc:2468] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:24.061857 1614 tablet_service.cc:1505] Processing CreateTablet for tablet a7e3150aea7f41af9e998ef0964b19a3 (DEFAULT_TABLE table=default.Table_02cc94782c6f48e5ae8aef6e9e879583 [id=bc634f121bac4d6295aad86e739795da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.064051 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1999b751e9344dcf81d26226c0589c8f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:24.064210 1510 raft_consensus.cc:3060] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:24.067067 1510 raft_consensus.cc:2468] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:24.068300 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "250f106664be4b5aaa330dbb4e43cb7b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:24.068462 1510 raft_consensus.cc:3060] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:24.069020 1586 consensus_peers.cc:597] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2a68bb1dc6c483388784b9bb19ce065. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:24.070073 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "250f106664be4b5aaa330dbb4e43cb7b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:24.071357 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1999b751e9344dcf81d26226c0589c8f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:24.072093 1510 raft_consensus.cc:2468] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
W20251024 19:48:24.071260 1586 consensus_peers.cc:597] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a1031d785b646b095ece9be666d2a77. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:24.073235 1186 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (table default.Table_18e7285b7df44399ab3be1dfc86eb91f [id=14bda7f9324f44aa9cb87a5f129039be]). Expected version 2 got 0
I20251024 19:48:24.076617 2858 raft_consensus.cc:493] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.076746 2858 raft_consensus.cc:515] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:24.077369 2858 leader_election.cc:290] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:24.078118 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "769f576810ab49368eb0fc7837b7da58" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:24.078718 1444 leader_election.cc:343] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.079156 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "769f576810ab49368eb0fc7837b7da58" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:24.079730 1447 leader_election.cc:343] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.079844 1447 leader_election.cc:304] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:24.080077 2858 raft_consensus.cc:2749] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:24.083356 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a5034103c1e4e039df7fc9f0ab65d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:24.084872 1472 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a5034103c1e4e039df7fc9f0ab65d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:24.088212 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7e3150aea7f41af9e998ef0964b19a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.088413 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e111980a15894b5084d9384297389b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.088591 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 043fac06e6324032a61c42a10d51ea22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.088762 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0276f82a2e78458f9d065c94c6d32edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.088943 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505ba3165385434faea556e97f42a7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.093739 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5282515c1844ae4b7469551436586c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.094199 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04dfa03ab59e4891ad5edd4003b9ceb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.095382 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 675301c87d784b0a8ffe04404960be58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.097759 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c70bfd01ae774245b42810722e508b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.099996 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ee0844646e4034b45f307738623968. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.102052 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c6fbc3067d433489230906816fc202. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.104322 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b7defa4cac4f62988eb0a0b1227997. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.105700 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08de859a187b4f978d1925607d2c6402. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.109692 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3440f176bd814230b220b32586cc1208. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.113714 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9379d71d772424891ed8d3d72cafd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.119534 1767 tablet_bootstrap.cc:654] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.121666 2858 raft_consensus.cc:493] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.121805 2858 raft_consensus.cc:515] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.122431 2858 leader_election.cc:290] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:24.123005 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292c31e7d4ce411bb336a1405ca43ba5" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:24.123400 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292c31e7d4ce411bb336a1405ca43ba5" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:24.123524 1447 leader_election.cc:343] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:24.123850 1444 leader_election.cc:343] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.123939 1444 leader_election.cc:304] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:24.124344 2858 raft_consensus.cc:2749] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:24.127230 1586 consensus_peers.cc:597] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d840888183ec4ef7bb245363c5bac322. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:24.127388 1586 leader_election.cc:343] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.127777 3404 consensus_queue.cc:1048] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:24.127940 1768 tablet_bootstrap.cc:654] T f965111ac60841dbb9e6ecc21ff3fec0 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:24.128429 1586 leader_election.cc:343] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:24.128829 1586 consensus_peers.cc:597] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:24.128971 1586 consensus_peers.cc:597] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:24.130714 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 5a67760335174b50baf6605a6bcc9d9b (DEFAULT_TABLE table=default.Table_200b6d5fd015449d877cd90dba49089f [id=12e0ac3fe51d430c8540cb19a42db706]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.131731 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a67760335174b50baf6605a6bcc9d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.133396 1776 ts_tablet_manager.cc:1929] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:24.133507 1776 log.cc:1199] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/a3259823b622461fa823841925b3d203
W20251024 19:48:24.133648 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f6f8cb6415d490184383c6f1ebe99f8 has no consensus state
I20251024 19:48:24.133889 1776 ts_tablet_manager.cc:1950] T a3259823b622461fa823841925b3d203 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:24.135097 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1036) took 3666 ms. Trace:
I20251024 19:48:24.135195 1776 rpcz_store.cc:276] 1024 19:48:20.468368 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.468441 (+ 73us) service_pool.cc:225] Handling call
1024 19:48:23.384118 (+2915677us) ts_tablet_manager.cc:1151] Deleting tablet a3259823b622461fa823841925b3d203
1024 19:48:23.384299 (+ 181us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.094769 (+710470us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.133307 (+ 38538us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.135077 (+ 1770us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2915277}
I20251024 19:48:24.135582 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet a3259823b622461fa823841925b3d203 (table default.Table_ea0d3589c13a4ebeb950ddc737a9948b [id=350b37bf42194fcfbb660bffe83e47c4]) successfully deleted
I20251024 19:48:24.135675 1776 tablet_replica.cc:333] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:24.135902 1776 raft_consensus.cc:2243] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:24.136261 1776 raft_consensus.cc:2272] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:24.136193 1186 catalog_manager.cc:5649] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } }
I20251024 19:48:24.136781 1776 ts_tablet_manager.cc:1916] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:24.057993 1774 ts_tablet_manager.cc:1929] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:24.138896 1774 log.cc:1199] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/f3d80885fcc642b8a75da30cd4abecbd
I20251024 19:48:24.139287 1774 ts_tablet_manager.cc:1950] T f3d80885fcc642b8a75da30cd4abecbd P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:24.139642 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1002) took 1415 ms. Trace:
I20251024 19:48:24.139745 1774 rpcz_store.cc:276] 1024 19:48:22.724516 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.724564 (+ 48us) service_pool.cc:225] Handling call
1024 19:48:23.865179 (+1140615us) ts_tablet_manager.cc:1151] Deleting tablet f3d80885fcc642b8a75da30cd4abecbd
1024 19:48:23.865192 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:23.967984 (+102792us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.057868 (+ 89884us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.139620 (+ 81752us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1140403}
I20251024 19:48:24.140091 1774 tablet_replica.cc:333] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:24.140287 1774 raft_consensus.cc:2243] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:24.140456 1774 raft_consensus.cc:2272] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:24.140790 1586 leader_election.cc:343] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:24.140990 1586 leader_election.cc:343] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:24.141676 1586 leader_election.cc:343] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.143185 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 33d42f4c818340cda4ae615132fcf03f
I20251024 19:48:24.144979 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 33d42f4c818340cda4ae615132fcf03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.146173 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 25af3192401b4666a8fecdc2f94af761 (DEFAULT_TABLE table=default.Table_9595525b6e594761ae90d7fcde6eaea4 [id=1777ec128e834d569a4301fb6ced2ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.146677 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25af3192401b4666a8fecdc2f94af761. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.153748 1766 tablet_bootstrap.cc:492] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:24.153786 1586 leader_election.cc:304] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:24.154016 1766 ts_tablet_manager.cc:1403] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.147s user 0.005s sys 0.003s
I20251024 19:48:24.154033 3404 raft_consensus.cc:2804] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:24.154098 3404 raft_consensus.cc:493] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:24.154170 3404 raft_consensus.cc:3060] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:24.154867 1766 raft_consensus.cc:359] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.155042 1766 raft_consensus.cc:385] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.155160 1766 raft_consensus.cc:740] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.155372 1766 consensus_queue.cc:260] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.156245 1766 ts_tablet_manager.cc:1434] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:24.156680 3404 raft_consensus.cc:515] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.157400 3404 leader_election.cc:290] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:24.157876 1586 leader_election.cc:304] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:24.158548 1586 leader_election.cc:304] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:24.158838 2923 raft_consensus.cc:2804] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:24.158964 2923 raft_consensus.cc:697] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:24.159250 2923 consensus_queue.cc:237] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.159770 3387 raft_consensus.cc:2804] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:24.159890 3387 raft_consensus.cc:697] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:24.160146 3387 consensus_queue.cc:237] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.161161 1345 tablet_service.cc:1505] Processing CreateTablet for tablet a7e3150aea7f41af9e998ef0964b19a3 (DEFAULT_TABLE table=default.Table_02cc94782c6f48e5ae8aef6e9e879583 [id=bc634f121bac4d6295aad86e739795da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.161649 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7e3150aea7f41af9e998ef0964b19a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.166625 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 331625f0a7d94884a62f08311fc06da5 (DEFAULT_TABLE table=default.Table_b39ea1cdfd63459c83c0f471a6c57015 [id=9761a4bb43064025a254f3a0cdedd83d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.167101 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331625f0a7d94884a62f08311fc06da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.169878 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 52d74c0a4e4743109114a446864c3ed6 (DEFAULT_TABLE table=default.Table_73c2345bb48f422ba47714dc6770bc3b [id=cd5a222c4eee4081a9721f2839e5df6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.161155 1768 tablet_bootstrap.cc:492] T f965111ac60841dbb9e6ecc21ff3fec0 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:24.170328 1768 ts_tablet_manager.cc:1403] T f965111ac60841dbb9e6ecc21ff3fec0 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.139s user 0.006s sys 0.000s
I20251024 19:48:24.170370 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52d74c0a4e4743109114a446864c3ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.171283 1768 raft_consensus.cc:359] T f965111ac60841dbb9e6ecc21ff3fec0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.171499 1768 raft_consensus.cc:385] T f965111ac60841dbb9e6ecc21ff3fec0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.171550 1768 raft_consensus.cc:740] T f965111ac60841dbb9e6ecc21ff3fec0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.171842 1768 consensus_queue.cc:260] T f965111ac60841dbb9e6ecc21ff3fec0 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.173280 1768 ts_tablet_manager.cc:1434] T f965111ac60841dbb9e6ecc21ff3fec0 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:24.173645 1768 tablet_bootstrap.cc:492] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:24.173902 1627 tablet_service.cc:1505] Processing CreateTablet for tablet f176a850e2fd4f79bbce0004a6c429e2 (DEFAULT_TABLE table=default.Table_ade70e87a32546219c5f6171be35ec6c [id=16719e8f101a48dcab8c7d525a4ba7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.174357 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f176a850e2fd4f79bbce0004a6c429e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.175724 1768 tablet_bootstrap.cc:654] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.177309 1472 tablet_service.cc:1505] Processing CreateTablet for tablet a7e3150aea7f41af9e998ef0964b19a3 (DEFAULT_TABLE table=default.Table_02cc94782c6f48e5ae8aef6e9e879583 [id=bc634f121bac4d6295aad86e739795da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.177845 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 1edc55caab904722bf19a1c71df601cf (table default.Table_641c979b063f419884a09e448c68a1b4 [id=b72bff5d78ec4eb08810442a084a16a8]) successfully deleted
I20251024 19:48:24.177848 1630 tablet_service.cc:1505] Processing CreateTablet for tablet c70e9097da534e4e995e43fb4e96e7f8 (DEFAULT_TABLE table=default.Table_498e11a28046450cbcd95b82f686f7de [id=990fa743890e4d41a0f1b2bc8abd50ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.178198 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c70e9097da534e4e995e43fb4e96e7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.178881 1774 ts_tablet_manager.cc:1916] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:24.181365 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 5a67760335174b50baf6605a6bcc9d9b (DEFAULT_TABLE table=default.Table_200b6d5fd015449d877cd90dba49089f [id=12e0ac3fe51d430c8540cb19a42db706]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.182391 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 5a67760335174b50baf6605a6bcc9d9b (DEFAULT_TABLE table=default.Table_200b6d5fd015449d877cd90dba49089f [id=12e0ac3fe51d430c8540cb19a42db706]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.182909 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a67760335174b50baf6605a6bcc9d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.182957 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 33d42f4c818340cda4ae615132fcf03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:24.185137 1775 ts_tablet_manager.cc:1929] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:24.185245 1775 log.cc:1199] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/96a6e0a4882c42399fd07f06ac21aa3f
I20251024 19:48:24.191354 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_dd7cc9b0637441719717ddd3eeafb570"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:24.191995 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd7cc9b0637441719717ddd3eeafb570 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:24.193341 3141 consensus_queue.cc:1048] T 385e71e672154e399f14321eac0d8625 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:24.193399 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_dcdcda89edd64b598f444eb9f4d98048"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:24.193921 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcdcda89edd64b598f444eb9f4d98048 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:24.205039 1584 connection.cc:537] client connection to 127.0.252.253:41375 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251024 19:48:24.205747 1306 connection.cc:537] client connection to 127.0.252.253:41375 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251024 19:48:24.206138 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7d7b19adc8404444ab1e322bac3adf0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:24.206722 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d7b19adc8404444ab1e322bac3adf0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:24.207305 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38818c7997c84cedad5f9b09eef77105" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:24.208859 1586 consensus_peers.cc:597] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:24.209000 1586 leader_election.cc:343] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:24.218381 1177 connection.cc:537] client connection to 127.0.252.253:41375 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251024 19:48:24.218538 1177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
W20251024 19:48:24.219691 3425 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 19:48:24.206105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.252.253:41375 (local address 127.0.252.195:40149)
1024 19:48:24.217821 (+ 11716us) negotiation.cc:107] Waiting for socket to connect
1024 19:48:24.217847 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1024 19:48:24.218011 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 19:48:24.219443 (+ 1432us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.252.253:41375: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":11562,"thread_start_us":12003,"threads_started":1}
W20251024 19:48:24.205449 1445 connection.cc:537] client connection to 127.0.252.253:41375 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251024 19:48:24.222143 1775 ts_tablet_manager.cc:1950] T 96a6e0a4882c42399fd07f06ac21aa3f P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:24.225649 1767 tablet_bootstrap.cc:492] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:24.225767 1767 ts_tablet_manager.cc:1403] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.440s user 0.007s sys 0.003s
I20251024 19:48:24.231747 1612 tablet_service.cc:1505] Processing CreateTablet for tablet de5981ed5284442b9d842a57e139500d (DEFAULT_TABLE table=default.Table_a3572e58f56043ca8b41124f7b194cee [id=0d21d2895f3c4b81b9fb5eb10a52b6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.232224 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de5981ed5284442b9d842a57e139500d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.233908 1767 raft_consensus.cc:359] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.234066 1767 raft_consensus.cc:385] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.234127 1767 raft_consensus.cc:740] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.234333 1767 consensus_queue.cc:260] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.234752 1767 ts_tablet_manager.cc:1434] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251024 19:48:24.235049 1767 tablet_bootstrap.cc:492] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:24.235100 1506 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38818c7997c84cedad5f9b09eef77105" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:24.235282 1506 raft_consensus.cc:3060] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:24.235728 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7e3150aea7f41af9e998ef0964b19a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.236435 1767 tablet_bootstrap.cc:654] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.244026 1506 raft_consensus.cc:2468] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:24.244501 1586 leader_election.cc:304] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:24.244982 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a67760335174b50baf6605a6bcc9d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.244957 3360 raft_consensus.cc:2804] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:24.245069 3360 raft_consensus.cc:697] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:24.245304 3360 consensus_queue.cc:237] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:24.251791 3427 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 19:48:24.208228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.252.253:41375 (local address 127.0.252.193:42799)
1024 19:48:24.251540 (+ 43312us) negotiation.cc:107] Waiting for socket to connect
1024 19:48:24.251638 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.252.253:41375: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":43176,"threads_started":1}
I20251024 19:48:24.252580 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1016) took 4167 ms. Trace:
I20251024 19:48:24.252743 1775 rpcz_store.cc:276] 1024 19:48:20.084686 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.084797 (+ 111us) service_pool.cc:225] Handling call
1024 19:48:24.045244 (+3960447us) ts_tablet_manager.cc:1151] Deleting tablet 96a6e0a4882c42399fd07f06ac21aa3f
1024 19:48:24.045254 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.111462 (+ 66208us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.185034 (+ 73572us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.252560 (+ 67526us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3960245}
I20251024 19:48:24.253243 1775 tablet_replica.cc:333] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:24.253484 1775 raft_consensus.cc:2243] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:24.253700 1775 raft_consensus.cc:2272] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:24.254307 1775 ts_tablet_manager.cc:1916] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:24.257365 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 8807f4dcb5df4588a99bfe15214c4128 (DEFAULT_TABLE table=default.Table_e3ba54a48d2c4fd594bbce560159e3aa [id=df80baf365d2427ca33be10129b074f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.257886 1766 tablet_bootstrap.cc:492] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:24.259491 1768 tablet_bootstrap.cc:492] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:24.258548 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 33d42f4c818340cda4ae615132fcf03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:24.263149 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8807f4dcb5df4588a99bfe15214c4128. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.263947 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 25af3192401b4666a8fecdc2f94af761 (DEFAULT_TABLE table=default.Table_9595525b6e594761ae90d7fcde6eaea4 [id=1777ec128e834d569a4301fb6ced2ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.264394 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25af3192401b4666a8fecdc2f94af761. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.260954 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 25af3192401b4666a8fecdc2f94af761 (DEFAULT_TABLE table=default.Table_9595525b6e594761ae90d7fcde6eaea4 [id=1777ec128e834d569a4301fb6ced2ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.272276 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25af3192401b4666a8fecdc2f94af761. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.261826 1774 ts_tablet_manager.cc:1929] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:24.279323 1774 log.cc:1199] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/28005301f9de48e5826f46c79aeb5fe5
I20251024 19:48:24.279835 1774 ts_tablet_manager.cc:1950] T 28005301f9de48e5826f46c79aeb5fe5 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:24.280853 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1001) took 1556 ms. Trace:
I20251024 19:48:24.281029 1774 rpcz_store.cc:276] 1024 19:48:22.724403 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.725698 (+ 1295us) service_pool.cc:225] Handling call
1024 19:48:24.140041 (+1414343us) ts_tablet_manager.cc:1151] Deleting tablet 28005301f9de48e5826f46c79aeb5fe5
1024 19:48:24.140049 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.239796 (+ 99747us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.261711 (+ 21915us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.280831 (+ 19120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1414149}
I20251024 19:48:24.281788 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1003) took 1546 ms. Trace:
I20251024 19:48:24.281935 1774 rpcz_store.cc:276] 1024 19:48:22.735579 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.735641 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:24.281651 (+1546010us) ts_tablet_manager.cc:1151] Deleting tablet 4e14a82420f24157a36df44d050b88bf
1024 19:48:24.281662 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.281771 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1545823}
I20251024 19:48:24.282397 1774 tablet_replica.cc:333] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:24.289857 1774 raft_consensus.cc:2243] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:24.290756 1472 tablet_service.cc:1505] Processing CreateTablet for tablet 8807f4dcb5df4588a99bfe15214c4128 (DEFAULT_TABLE table=default.Table_e3ba54a48d2c4fd594bbce560159e3aa [id=df80baf365d2427ca33be10129b074f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.291239 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8807f4dcb5df4588a99bfe15214c4128. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.260027 1613 tablet_service.cc:1505] Processing CreateTablet for tablet d8ee532293634fa0b824b2ec222545b9 (DEFAULT_TABLE table=default.Table_e19e1939816b40ddb5131877638d430b [id=f66f6d31f481471abfe92f1589a7c06a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.290889 1767 tablet_bootstrap.cc:492] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:24.299501 1767 ts_tablet_manager.cc:1403] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.001s
I20251024 19:48:24.299826 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ee532293634fa0b824b2ec222545b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.300308 1767 raft_consensus.cc:359] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.300503 1767 raft_consensus.cc:385] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.300570 1767 raft_consensus.cc:740] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.283329 1344 tablet_service.cc:1505] Processing CreateTablet for tablet 8807f4dcb5df4588a99bfe15214c4128 (DEFAULT_TABLE table=default.Table_e3ba54a48d2c4fd594bbce560159e3aa [id=df80baf365d2427ca33be10129b074f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.282621 1768 ts_tablet_manager.cc:1403] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.109s user 0.004s sys 0.000s
I20251024 19:48:24.300884 1767 consensus_queue.cc:260] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.300762 1340 tablet_service.cc:1505] Processing CreateTablet for tablet f176a850e2fd4f79bbce0004a6c429e2 (DEFAULT_TABLE table=default.Table_ade70e87a32546219c5f6171be35ec6c [id=16719e8f101a48dcab8c7d525a4ba7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.301466 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8807f4dcb5df4588a99bfe15214c4128. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.301733 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f176a850e2fd4f79bbce0004a6c429e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.259522 1776 ts_tablet_manager.cc:1929] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:24.303433 1776 log.cc:1199] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/9a1031d785b646b095ece9be666d2a77
I20251024 19:48:24.303845 1776 ts_tablet_manager.cc:1950] T 9a1031d785b646b095ece9be666d2a77 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:24.304597 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1037) took 3790 ms. Trace:
I20251024 19:48:24.304678 1776 rpcz_store.cc:276] 1024 19:48:20.514138 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.514202 (+ 64us) service_pool.cc:225] Handling call
1024 19:48:24.135617 (+3621415us) ts_tablet_manager.cc:1151] Deleting tablet 9a1031d785b646b095ece9be666d2a77
1024 19:48:24.135628 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.232169 (+ 96541us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.259406 (+ 27237us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.304574 (+ 45168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3621182}
I20251024 19:48:24.305238 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1038) took 3779 ms. Trace:
I20251024 19:48:24.305315 1776 rpcz_store.cc:276] 1024 19:48:20.525366 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.525443 (+ 77us) service_pool.cc:225] Handling call
1024 19:48:24.305102 (+3779659us) ts_tablet_manager.cc:1151] Deleting tablet a2cc3f3d02324c1e867b8fc159ad6d20
1024 19:48:24.305114 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.305220 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3779398}
I20251024 19:48:24.305768 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1046) took 3189 ms. Trace:
I20251024 19:48:24.305832 1776 rpcz_store.cc:276] 1024 19:48:21.116109 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.116165 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:24.305668 (+3189503us) ts_tablet_manager.cc:1151] Deleting tablet 04ae62c9bf6f44108d57d0760283d063
1024 19:48:24.305677 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.305753 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3189292}
I20251024 19:48:24.306243 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1045) took 3190 ms. Trace:
I20251024 19:48:24.306306 1776 rpcz_store.cc:276] 1024 19:48:21.115928 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.116172 (+ 244us) service_pool.cc:225] Handling call
1024 19:48:24.306111 (+3189939us) ts_tablet_manager.cc:1151] Deleting tablet 7ecf3276437a43cf84a47a0e66553e0a
1024 19:48:24.306119 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.306230 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3189650}
I20251024 19:48:24.306684 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1049) took 3127 ms. Trace:
I20251024 19:48:24.306758 1776 rpcz_store.cc:276] 1024 19:48:21.178938 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.178992 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:24.306593 (+3127601us) ts_tablet_manager.cc:1151] Deleting tablet e0b57403cdf947b390a44a9889f2a2c3
1024 19:48:24.306602 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.306669 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3127421}
I20251024 19:48:24.307127 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1048) took 3128 ms. Trace:
I20251024 19:48:24.307262 1776 rpcz_store.cc:276] 1024 19:48:21.178827 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.179337 (+ 510us) service_pool.cc:225] Handling call
1024 19:48:24.307041 (+3127704us) ts_tablet_manager.cc:1151] Deleting tablet 672b571081ea4cd78028b6a25625103c
1024 19:48:24.307050 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.307114 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3127531}
I20251024 19:48:24.307787 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1047) took 3129 ms. Trace:
I20251024 19:48:24.307919 1776 rpcz_store.cc:276] 1024 19:48:21.178698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.179587 (+ 889us) service_pool.cc:225] Handling call
1024 19:48:24.307688 (+3128101us) ts_tablet_manager.cc:1151] Deleting tablet 30bce965315d43948ecb8282c829c557
1024 19:48:24.307697 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.307770 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3127956}
I20251024 19:48:24.308413 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1054) took 3108 ms. Trace:
I20251024 19:48:24.308424 1774 raft_consensus.cc:2272] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:24.308570 1776 rpcz_store.cc:276] 1024 19:48:21.200394 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.200458 (+ 64us) service_pool.cc:225] Handling call
1024 19:48:24.308304 (+3107846us) ts_tablet_manager.cc:1151] Deleting tablet 3a99e9b3fe044c7eb14fa9bffa903873
1024 19:48:24.308313 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.308396 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3107654}
I20251024 19:48:24.309340 1776 tablet_replica.cc:333] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:24.309419 1475 tablet_service.cc:1505] Processing CreateTablet for tablet f176a850e2fd4f79bbce0004a6c429e2 (DEFAULT_TABLE table=default.Table_ade70e87a32546219c5f6171be35ec6c [id=16719e8f101a48dcab8c7d525a4ba7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.309638 1776 raft_consensus.cc:2243] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:24.309875 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f176a850e2fd4f79bbce0004a6c429e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.310765 1767 ts_tablet_manager.cc:1434] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251024 19:48:24.311120 1767 tablet_bootstrap.cc:492] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:24.312497 1767 tablet_bootstrap.cc:654] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.316591 1768 raft_consensus.cc:359] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.323401 1766 tablet_bootstrap.cc:654] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.322285 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 331625f0a7d94884a62f08311fc06da5 (DEFAULT_TABLE table=default.Table_b39ea1cdfd63459c83c0f471a6c57015 [id=9761a4bb43064025a254f3a0cdedd83d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.325565 1768 raft_consensus.cc:385] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.322523 1774 ts_tablet_manager.cc:1916] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:24.325764 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331625f0a7d94884a62f08311fc06da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.325867 1768 raft_consensus.cc:740] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.331300 1768 consensus_queue.cc:260] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.328975 2858 raft_consensus.cc:493] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.331841 2858 raft_consensus.cc:515] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.332587 2858 leader_election.cc:290] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:24.333053 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da7d38b63708427982591da1014aca81" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:24.333212 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da7d38b63708427982591da1014aca81" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:24.333540 1447 leader_election.cc:343] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:24.333760 1444 leader_election.cc:343] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.333864 1444 leader_election.cc:304] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:24.334201 2858 raft_consensus.cc:2749] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:24.338117 1768 ts_tablet_manager.cc:1434] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20251024 19:48:24.338596 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 9a1031d785b646b095ece9be666d2a77 (table default.Table_a1d09325af844d71b290d95b8a4aaf12 [id=b62ec7bcad7c41b1b946e9ebf2f4bf39]) successfully deleted
W20251024 19:48:24.339118 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet a2cc3f3d02324c1e867b8fc159ad6d20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2cc3f3d02324c1e867b8fc159ad6d20 already in progress: creating tablet
W20251024 19:48:24.340389 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 04ae62c9bf6f44108d57d0760283d063 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04ae62c9bf6f44108d57d0760283d063 already in progress: creating tablet
I20251024 19:48:24.342240 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 52d74c0a4e4743109114a446864c3ed6 (DEFAULT_TABLE table=default.Table_73c2345bb48f422ba47714dc6770bc3b [id=cd5a222c4eee4081a9721f2839e5df6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.342294 1475 tablet_service.cc:1505] Processing CreateTablet for tablet c70e9097da534e4e995e43fb4e96e7f8 (DEFAULT_TABLE table=default.Table_498e11a28046450cbcd95b82f686f7de [id=990fa743890e4d41a0f1b2bc8abd50ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.342672 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52d74c0a4e4743109114a446864c3ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.342993 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c70e9097da534e4e995e43fb4e96e7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.350450 1472 tablet_service.cc:1505] Processing CreateTablet for tablet 331625f0a7d94884a62f08311fc06da5 (DEFAULT_TABLE table=default.Table_b39ea1cdfd63459c83c0f471a6c57015 [id=9761a4bb43064025a254f3a0cdedd83d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.351166 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331625f0a7d94884a62f08311fc06da5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:24.351327 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7ecf3276437a43cf84a47a0e66553e0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ecf3276437a43cf84a47a0e66553e0a already in progress: creating tablet
I20251024 19:48:24.353252 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 4b4904080b7a449fb92e773032808533 (DEFAULT_TABLE table=default.Table_2691f5c07d014467892d86561d71800b [id=e3f90a94f29f44ee9741413bda25df9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.353775 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4904080b7a449fb92e773032808533. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.341809 1351 tablet_service.cc:1505] Processing CreateTablet for tablet de5981ed5284442b9d842a57e139500d (DEFAULT_TABLE table=default.Table_a3572e58f56043ca8b41124f7b194cee [id=0d21d2895f3c4b81b9fb5eb10a52b6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.358016 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de5981ed5284442b9d842a57e139500d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:24.361449 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet e0b57403cdf947b390a44a9889f2a2c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0b57403cdf947b390a44a9889f2a2c3 already in progress: creating tablet
W20251024 19:48:24.361836 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 672b571081ea4cd78028b6a25625103c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 672b571081ea4cd78028b6a25625103c already in progress: creating tablet
W20251024 19:48:24.362217 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 30bce965315d43948ecb8282c829c557 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30bce965315d43948ecb8282c829c557 already in progress: creating tablet
W20251024 19:48:24.362443 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3a99e9b3fe044c7eb14fa9bffa903873 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a99e9b3fe044c7eb14fa9bffa903873 already in progress: creating tablet
I20251024 19:48:24.363040 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet f3d80885fcc642b8a75da30cd4abecbd (table default.Table_7966ff9879d24254aed39562b2f127d7 [id=45b38d5e9fd6472fa318b41863080708]) successfully deleted
I20251024 19:48:24.363636 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 3a2d631f868e488bb4105566d374d041 (DEFAULT_TABLE table=default.Table_a9b2002bb5a544c698822168e561c8cd [id=7634538840c4416ea708f5826e11c6a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.364019 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2d631f868e488bb4105566d374d041. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.364056 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 8f46974e56c64a58a9c562b235ff8cfe (DEFAULT_TABLE table=default.Table_4f22d5090caa447d9715e95a8a248432 [id=b9749925c69f4200888dcfe96f2b2e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.364524 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f46974e56c64a58a9c562b235ff8cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.345639 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 52d74c0a4e4743109114a446864c3ed6 (DEFAULT_TABLE table=default.Table_73c2345bb48f422ba47714dc6770bc3b [id=cd5a222c4eee4081a9721f2839e5df6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.364912 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52d74c0a4e4743109114a446864c3ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.369925 1620 tablet_service.cc:1505] Processing CreateTablet for tablet cb614fcf24714b7da1485d7f13d57ee4 (DEFAULT_TABLE table=default.Table_160ec7727d65458cbbf48a790008ba6b [id=1b48afe796ef4c9fac83402ad1a357b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.370421 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb614fcf24714b7da1485d7f13d57ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.370370 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 62e987e72ac94f1b9716b3a3644eed83 (DEFAULT_TABLE table=default.Table_c93f1a18914a43839b3bf865dd238a1b [id=1ee14fbcc12348b09b9f645e99d7dd4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.370723 1624 tablet_service.cc:1505] Processing CreateTablet for tablet ec7ad30bbf83424a8ad4d4383e0e56da (DEFAULT_TABLE table=default.Table_3ddc38c30d594d45bc5e37c0994e4dde [id=0d1b6563a39a4f4a902e423cb8551d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.352309 1774 ts_tablet_manager.cc:1929] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:24.371116 1774 log.cc:1199] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/59956dad06064024ab171631ef6c1dca
I20251024 19:48:24.371193 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62e987e72ac94f1b9716b3a3644eed83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.371482 1774 ts_tablet_manager.cc:1950] T 59956dad06064024ab171631ef6c1dca P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:24.372311 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1004) took 1432 ms. Trace:
I20251024 19:48:24.372391 1774 rpcz_store.cc:276] 1024 19:48:22.939384 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.939588 (+ 204us) service_pool.cc:225] Handling call
1024 19:48:24.282348 (+1342760us) ts_tablet_manager.cc:1151] Deleting tablet 59956dad06064024ab171631ef6c1dca
1024 19:48:24.282356 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.337286 (+ 54930us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.352202 (+ 14916us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.372287 (+ 20085us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1342537}
I20251024 19:48:24.372809 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1005) took 1433 ms. Trace:
I20251024 19:48:24.372879 1774 rpcz_store.cc:276] 1024 19:48:22.939781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.939835 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:24.372685 (+1432850us) ts_tablet_manager.cc:1151] Deleting tablet 4e14a82420f24157a36df44d050b88bf
1024 19:48:24.372693 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.372794 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1432714}
I20251024 19:48:24.373163 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1022) took 1259 ms. Trace:
I20251024 19:48:24.373222 1774 rpcz_store.cc:276] 1024 19:48:23.114069 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.114120 (+ 51us) service_pool.cc:225] Handling call
1024 19:48:24.373074 (+1258954us) ts_tablet_manager.cc:1151] Deleting tablet 80688d7b8a394a2a910c4b8910fe4cf2
1024 19:48:24.373084 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.373150 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1258686}
I20251024 19:48:24.373471 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1024) took 1259 ms. Trace:
I20251024 19:48:24.373639 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 28005301f9de48e5826f46c79aeb5fe5 (table default.Table_e74966425ce8408da939910bc645012d [id=e345af66b89e48f9afe96000ce50179c]) successfully deleted
I20251024 19:48:24.373765 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec7ad30bbf83424a8ad4d4383e0e56da. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:24.378540 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4e14a82420f24157a36df44d050b88bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e14a82420f24157a36df44d050b88bf already in progress: creating tablet
I20251024 19:48:24.373534 1774 rpcz_store.cc:276] 1024 19:48:23.114252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.114300 (+ 48us) service_pool.cc:225] Handling call
1024 19:48:24.373394 (+1259094us) ts_tablet_manager.cc:1151] Deleting tablet 9198321bcc9e4e47a4c2a5442b6bc850
1024 19:48:24.373401 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.373459 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1258940}
I20251024 19:48:24.379086 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1023) took 1264 ms. Trace:
I20251024 19:48:24.379359 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 894b9d58376f45c79c96a75b6aa3c6b0 (DEFAULT_TABLE table=default.Table_64e7af7019f24a8eaf186a4e93208f03 [id=0dff97dda04e486483da5757b04c6ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.379511 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 59956dad06064024ab171631ef6c1dca (table default.Table_9c03ce6b892d44fcbff5c92f613bc37f [id=e51bd6230e2e41429af7e8b08198bbe1]) successfully deleted
I20251024 19:48:24.379963 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894b9d58376f45c79c96a75b6aa3c6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.381192 1774 rpcz_store.cc:276] 1024 19:48:23.114167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.114311 (+ 144us) service_pool.cc:225] Handling call
1024 19:48:24.378950 (+1264639us) ts_tablet_manager.cc:1151] Deleting tablet 83d34ec47c2845058e870c72cc832e9e
1024 19:48:24.378961 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.379066 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1264341}
I20251024 19:48:24.381793 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1025) took 1267 ms. Trace:
I20251024 19:48:24.381906 1774 rpcz_store.cc:276] 1024 19:48:23.114571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.114612 (+ 41us) service_pool.cc:225] Handling call
1024 19:48:24.381645 (+1267033us) ts_tablet_manager.cc:1151] Deleting tablet 9b7ab1bf5b9549afb150802a209527eb
1024 19:48:24.381657 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.381773 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266893}
I20251024 19:48:24.382210 1774 tablet_replica.cc:333] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:24.382434 1774 raft_consensus.cc:2243] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
W20251024 19:48:24.383574 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4e14a82420f24157a36df44d050b88bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e14a82420f24157a36df44d050b88bf already in progress: creating tablet
I20251024 19:48:24.383915 1774 pending_rounds.cc:70] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:24.383994 1774 pending_rounds.cc:77] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429404467384320 op_type: NO_OP noop_request { }
I20251024 19:48:24.383631 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 7b4eb709e46845428e6ed746aa6d0f57 (DEFAULT_TABLE table=default.Table_0c171a897bcc47e5a625cfbb07056e74 [id=d6f1df1c2ece4303888bd3e6d35dcd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.384282 1774 raft_consensus.cc:2889] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:24.384385 1774 raft_consensus.cc:2272] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:24.384720 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b4eb709e46845428e6ed746aa6d0f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.385545 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 3fb5537dcb424075bea484cfd6b33766 (DEFAULT_TABLE table=default.Table_68e567e80b54411a9ff4a53e1e8961a6 [id=18b6a9163f3b4d66b189781ca6cf6202]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.385944 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fb5537dcb424075bea484cfd6b33766. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:24.386230 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 80688d7b8a394a2a910c4b8910fe4cf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80688d7b8a394a2a910c4b8910fe4cf2 already in progress: creating tablet
W20251024 19:48:24.386567 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9198321bcc9e4e47a4c2a5442b6bc850 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9198321bcc9e4e47a4c2a5442b6bc850 already in progress: creating tablet
W20251024 19:48:24.386776 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 83d34ec47c2845058e870c72cc832e9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83d34ec47c2845058e870c72cc832e9e already in progress: creating tablet
W20251024 19:48:24.387177 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9b7ab1bf5b9549afb150802a209527eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b7ab1bf5b9549afb150802a209527eb already in progress: creating tablet
I20251024 19:48:24.390082 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 8f46974e56c64a58a9c562b235ff8cfe (DEFAULT_TABLE table=default.Table_4f22d5090caa447d9715e95a8a248432 [id=b9749925c69f4200888dcfe96f2b2e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.390620 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f46974e56c64a58a9c562b235ff8cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.391742 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 4b4904080b7a449fb92e773032808533 (DEFAULT_TABLE table=default.Table_2691f5c07d014467892d86561d71800b [id=e3f90a94f29f44ee9741413bda25df9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.392067 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4904080b7a449fb92e773032808533. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.393842 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 7b4eb709e46845428e6ed746aa6d0f57 (DEFAULT_TABLE table=default.Table_0c171a897bcc47e5a625cfbb07056e74 [id=d6f1df1c2ece4303888bd3e6d35dcd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.394160 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b4eb709e46845428e6ed746aa6d0f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.395659 1349 tablet_service.cc:1505] Processing CreateTablet for tablet cb614fcf24714b7da1485d7f13d57ee4 (DEFAULT_TABLE table=default.Table_160ec7727d65458cbbf48a790008ba6b [id=1b48afe796ef4c9fac83402ad1a357b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.395988 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb614fcf24714b7da1485d7f13d57ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.397671 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 3fb5537dcb424075bea484cfd6b33766 (DEFAULT_TABLE table=default.Table_68e567e80b54411a9ff4a53e1e8961a6 [id=18b6a9163f3b4d66b189781ca6cf6202]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.397974 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fb5537dcb424075bea484cfd6b33766. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.399103 1336 tablet_service.cc:1505] Processing CreateTablet for tablet ec7ad30bbf83424a8ad4d4383e0e56da (DEFAULT_TABLE table=default.Table_3ddc38c30d594d45bc5e37c0994e4dde [id=0d1b6563a39a4f4a902e423cb8551d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.399406 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec7ad30bbf83424a8ad4d4383e0e56da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.399878 1774 ts_tablet_manager.cc:1916] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:24.401021 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 96a6e0a4882c42399fd07f06ac21aa3f (table default.Table_19bbc037a7f241f5b338f4cfa4ebe82d [id=b2ce46e067994c9fa4df9f00de59d68f]) successfully deleted
I20251024 19:48:24.401831 1352 tablet_service.cc:1505] Processing CreateTablet for tablet 62e987e72ac94f1b9716b3a3644eed83 (DEFAULT_TABLE table=default.Table_c93f1a18914a43839b3bf865dd238a1b [id=1ee14fbcc12348b09b9f645e99d7dd4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.402144 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62e987e72ac94f1b9716b3a3644eed83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.404757 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 33cfa2e0bda94d7db34020f04688679d (DEFAULT_TABLE table=default.Table_48d22723ce864c7bbd947ce17e8e02f3 [id=a78ec11421af4e1391662f1f0f0f0195]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.405019 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 7b503fe19ecc44ee8086855d748b36aa (DEFAULT_TABLE table=default.Table_155f8d10b0d84715a1308d35a0d2b33a [id=be649bc161114b59a47492ef5afd3bb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.405246 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33cfa2e0bda94d7db34020f04688679d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.405351 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 894b9d58376f45c79c96a75b6aa3c6b0 (DEFAULT_TABLE table=default.Table_64e7af7019f24a8eaf186a4e93208f03 [id=0dff97dda04e486483da5757b04c6ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.405743 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894b9d58376f45c79c96a75b6aa3c6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.405573 1627 tablet_service.cc:1505] Processing CreateTablet for tablet 9afc93ecc49347f196545d5358db3fe8 (DEFAULT_TABLE table=default.Table_5899d327ab3844daa60d50f0035a4751 [id=65e76e5db67546ddb669e3d193ee541b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.406363 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9afc93ecc49347f196545d5358db3fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.409372 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 045fab6bf9f549f2bede99c518508a3d (DEFAULT_TABLE table=default.Table_cd1a44e3831a434fa5c98641671fba48 [id=7b07845333914faa9dd73a8d32a11c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.352748 1344 tablet_service.cc:1505] Processing CreateTablet for tablet c70e9097da534e4e995e43fb4e96e7f8 (DEFAULT_TABLE table=default.Table_498e11a28046450cbcd95b82f686f7de [id=990fa743890e4d41a0f1b2bc8abd50ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.409762 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 045fab6bf9f549f2bede99c518508a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.405840 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 7a95dbd3fdb241369e1bb4ebc72bc91c (DEFAULT_TABLE table=default.Table_3a4994ec76cc4afaa6e9bb2a0fa8a353 [id=d83692ef452047daa39a765057bb704b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.388401 1483 tablet_service.cc:1505] Processing CreateTablet for tablet d8ee532293634fa0b824b2ec222545b9 (DEFAULT_TABLE table=default.Table_e19e1939816b40ddb5131877638d430b [id=f66f6d31f481471abfe92f1589a7c06a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.410079 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a95dbd3fdb241369e1bb4ebc72bc91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.410120 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ee532293634fa0b824b2ec222545b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.388770 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 8f46974e56c64a58a9c562b235ff8cfe (DEFAULT_TABLE table=default.Table_4f22d5090caa447d9715e95a8a248432 [id=b9749925c69f4200888dcfe96f2b2e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.406020 1611 tablet_service.cc:1505] Processing CreateTablet for tablet aa55ecf90e3141fba598df64343c0f6e (DEFAULT_TABLE table=default.Table_9108635a402d40dba89df9a92ae028b7 [id=8c070d1f7a454c6aba743f27624c92c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.413951 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f46974e56c64a58a9c562b235ff8cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.413983 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa55ecf90e3141fba598df64343c0f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.414021 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 3a2d631f868e488bb4105566d374d041 (DEFAULT_TABLE table=default.Table_a9b2002bb5a544c698822168e561c8cd [id=7634538840c4416ea708f5826e11c6a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.414366 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2d631f868e488bb4105566d374d041. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.417290 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b503fe19ecc44ee8086855d748b36aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.391095 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 4b4904080b7a449fb92e773032808533 (DEFAULT_TABLE table=default.Table_2691f5c07d014467892d86561d71800b [id=e3f90a94f29f44ee9741413bda25df9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.417936 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4904080b7a449fb92e773032808533. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.391393 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 7b4eb709e46845428e6ed746aa6d0f57 (DEFAULT_TABLE table=default.Table_0c171a897bcc47e5a625cfbb07056e74 [id=d6f1df1c2ece4303888bd3e6d35dcd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.395740 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 3a2d631f868e488bb4105566d374d041 (DEFAULT_TABLE table=default.Table_a9b2002bb5a544c698822168e561c8cd [id=7634538840c4416ea708f5826e11c6a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.421831 3387 raft_consensus.cc:493] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.396350 1486 tablet_service.cc:1505] Processing CreateTablet for tablet cb614fcf24714b7da1485d7f13d57ee4 (DEFAULT_TABLE table=default.Table_160ec7727d65458cbbf48a790008ba6b [id=1b48afe796ef4c9fac83402ad1a357b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.421952 3387 raft_consensus.cc:515] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.396662 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 62e987e72ac94f1b9716b3a3644eed83 (DEFAULT_TABLE table=default.Table_c93f1a18914a43839b3bf865dd238a1b [id=1ee14fbcc12348b09b9f645e99d7dd4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.397315 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 894b9d58376f45c79c96a75b6aa3c6b0 (DEFAULT_TABLE table=default.Table_64e7af7019f24a8eaf186a4e93208f03 [id=0dff97dda04e486483da5757b04c6ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.397806 1473 tablet_service.cc:1505] Processing CreateTablet for tablet ec7ad30bbf83424a8ad4d4383e0e56da (DEFAULT_TABLE table=default.Table_3ddc38c30d594d45bc5e37c0994e4dde [id=0d1b6563a39a4f4a902e423cb8551d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.422607 3387 leader_election.cc:290] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:24.423141 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee5356b5503c4425bf6648ade5f6330e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:24.423210 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee5356b5503c4425bf6648ade5f6330e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:24.423334 1509 raft_consensus.cc:2468] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:24.423375 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2d631f868e488bb4105566d374d041. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.423573 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62e987e72ac94f1b9716b3a3644eed83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.423689 2858 raft_consensus.cc:493] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.423804 2858 raft_consensus.cc:515] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.424326 2858 leader_election.cc:290] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:24.424860 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b632acc1348e49f8a1eea019c007780c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:24.425309 1351 tablet_service.cc:1505] Processing CreateTablet for tablet d8ee532293634fa0b824b2ec222545b9 (DEFAULT_TABLE table=default.Table_e19e1939816b40ddb5131877638d430b [id=f66f6d31f481471abfe92f1589a7c06a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.425683 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ee532293634fa0b824b2ec222545b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:24.428211 1444 leader_election.cc:343] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.429625 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec7ad30bbf83424a8ad4d4383e0e56da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.429631 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b632acc1348e49f8a1eea019c007780c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:24.399782 1487 tablet_service.cc:1505] Processing CreateTablet for tablet de5981ed5284442b9d842a57e139500d (DEFAULT_TABLE table=default.Table_a3572e58f56043ca8b41124f7b194cee [id=0d21d2895f3c4b81b9fb5eb10a52b6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.402863 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 3fb5537dcb424075bea484cfd6b33766 (DEFAULT_TABLE table=default.Table_68e567e80b54411a9ff4a53e1e8961a6 [id=18b6a9163f3b4d66b189781ca6cf6202]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.429944 1474 tablet_service.cc:1552] Processing DeleteTablet for tablet 292c31e7d4ce411bb336a1405ca43ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:24.430179 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fb5537dcb424075bea484cfd6b33766. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.433741 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894b9d58376f45c79c96a75b6aa3c6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.436312 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de5981ed5284442b9d842a57e139500d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.437672 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb614fcf24714b7da1485d7f13d57ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.440495 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 045fab6bf9f549f2bede99c518508a3d (DEFAULT_TABLE table=default.Table_cd1a44e3831a434fa5c98641671fba48 [id=7b07845333914faa9dd73a8d32a11c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.440886 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 045fab6bf9f549f2bede99c518508a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.441672 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b4eb709e46845428e6ed746aa6d0f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.358031 1768 tablet_bootstrap.cc:492] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:24.445811 1586 leader_election.cc:343] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.446026 1586 leader_election.cc:304] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:24.446264 3387 raft_consensus.cc:2804] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:24.446337 3387 raft_consensus.cc:493] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:24.446395 3387 raft_consensus.cc:3060] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:24.447131 1447 leader_election.cc:343] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.447245 1447 leader_election.cc:304] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:24.447372 1768 tablet_bootstrap.cc:654] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.447494 2858 raft_consensus.cc:2749] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:24.447726 2858 raft_consensus.cc:493] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.447821 2858 raft_consensus.cc:515] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.448341 2858 leader_election.cc:290] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:24.448828 3387 raft_consensus.cc:515] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.448894 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83b91a5d764449d5b82ef6d1ee0360f5" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:24.449430 3387 leader_election.cc:290] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:24.449429 1444 leader_election.cc:343] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.449848 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee5356b5503c4425bf6648ade5f6330e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:24.449839 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee5356b5503c4425bf6648ade5f6330e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:24.449980 1509 raft_consensus.cc:3060] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:24.450165 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83b91a5d764449d5b82ef6d1ee0360f5" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:24.453083 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c70e9097da534e4e995e43fb4e96e7f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:24.453118 1586 leader_election.cc:343] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:24.453784 1447 leader_election.cc:343] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.453889 1447 leader_election.cc:304] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:24.453958 1776 raft_consensus.cc:2272] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:24.454116 3390 raft_consensus.cc:2749] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:24.455070 1776 ts_tablet_manager.cc:1916] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:24.455649 1642 raft_consensus.cc:1275] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 2 FOLLOWER]: Refusing update from remote peer 43bec2f03fc849099e254f6335e9d532: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20251024 19:48:24.455734 1642 pending_rounds.cc:85] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97: Aborting all ops after (but not including) 0
I20251024 19:48:24.455780 1642 pending_rounds.cc:107] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20251024 19:48:24.455863 1642 raft_consensus.cc:2889] T a1fbbba88a294adfbf97c0343d86698e P af3952de050140d383f6b0de7666fa97 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251024 19:48:24.456691 1509 raft_consensus.cc:2468] T ee5356b5503c4425bf6648ade5f6330e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
W20251024 19:48:24.457677 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 83830.6us
I20251024 19:48:24.457765 1586 leader_election.cc:304] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:24.458030 1766 tablet_bootstrap.cc:492] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:24.458077 3387 raft_consensus.cc:2804] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:24.458164 1766 ts_tablet_manager.cc:1403] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.200s user 0.003s sys 0.000s
I20251024 19:48:24.458184 3387 raft_consensus.cc:697] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:24.458413 3387 consensus_queue.cc:237] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.460464 1767 tablet_bootstrap.cc:492] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:24.460835 1767 ts_tablet_manager.cc:1403] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.150s user 0.001s sys 0.002s
I20251024 19:48:24.461861 3396 consensus_queue.cc:1048] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:24.463471 1767 raft_consensus.cc:359] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.464018 1767 raft_consensus.cc:385] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.464223 1767 raft_consensus.cc:740] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.464596 1767 consensus_queue.cc:260] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:24.465718 1177 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.253:41375, user_credentials={real_user=slave}} blocked reactor thread for 247127us
I20251024 19:48:24.465849 1767 ts_tablet_manager.cc:1434] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:48:24.468832 1611 tablet_service.cc:1505] Processing CreateTablet for tablet a52df993b09d43eeab7ffa9144ec6289 (DEFAULT_TABLE table=default.Table_cd3855acb8cc4a7486db5bbc7d37a26e [id=487680c59db34cb8be159611a56895b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.469405 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52df993b09d43eeab7ffa9144ec6289. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.471447 1627 tablet_service.cc:1505] Processing CreateTablet for tablet f73069b9201449d08a4f7b246444e891 (DEFAULT_TABLE table=default.Table_3b59a05b195a4d728ae60c0e25c5037f [id=3d9bfe7a3a90415980fc639c42d38e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.471832 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73069b9201449d08a4f7b246444e891. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.472580 1625 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54246 (request call id 1117) took 1128 ms. Trace:
I20251024 19:48:24.472651 1625 rpcz_store.cc:276] 1024 19:48:23.343917 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.343978 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:23.344667 (+ 689us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1024 19:48:23.344696 (+ 29us) ts_tablet_manager.cc:682] Creating new metadata...
1024 19:48:24.287374 (+942678us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.472424 (+185050us) tablet_replica.cc:206] Creating consensus instance
1024 19:48:24.472566 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251024 19:48:24.474035 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 7612aae4a39a41d781a2b654b2999b32 (DEFAULT_TABLE table=default.Table_7c6e82a967ab492ebf7d644969562282 [id=f61209500a4941279a01276631d3b1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.474409 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7612aae4a39a41d781a2b654b2999b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.476352 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 045fab6bf9f549f2bede99c518508a3d (DEFAULT_TABLE table=default.Table_cd1a44e3831a434fa5c98641671fba48 [id=7b07845333914faa9dd73a8d32a11c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.476715 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 045fab6bf9f549f2bede99c518508a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.479102 1775 ts_tablet_manager.cc:1929] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:24.479211 1775 log.cc:1199] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/65424b00ab9c454f9584048caba83e6a
I20251024 19:48:24.479509 1775 ts_tablet_manager.cc:1950] T 65424b00ab9c454f9584048caba83e6a P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:24.482586 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 33cfa2e0bda94d7db34020f04688679d (DEFAULT_TABLE table=default.Table_48d22723ce864c7bbd947ce17e8e02f3 [id=a78ec11421af4e1391662f1f0f0f0195]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.482999 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33cfa2e0bda94d7db34020f04688679d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.484153 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 33cfa2e0bda94d7db34020f04688679d (DEFAULT_TABLE table=default.Table_48d22723ce864c7bbd947ce17e8e02f3 [id=a78ec11421af4e1391662f1f0f0f0195]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.484607 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33cfa2e0bda94d7db34020f04688679d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.486383 1766 raft_consensus.cc:359] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.489941 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 02e6ab4d86954cb6b00376cc8d52e3c6 (DEFAULT_TABLE table=default.Table_9fcc7f42f92641fdbef94211601c4a32 [id=7519e13af9084a5d8ff34dc84c27f4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.490367 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e6ab4d86954cb6b00376cc8d52e3c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.493615 1766 raft_consensus.cc:385] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.490887 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 9afc93ecc49347f196545d5358db3fe8 (DEFAULT_TABLE table=default.Table_5899d327ab3844daa60d50f0035a4751 [id=65e76e5db67546ddb669e3d193ee541b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.493710 1766 raft_consensus.cc:740] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.493980 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9afc93ecc49347f196545d5358db3fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.493922 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 7612aae4a39a41d781a2b654b2999b32 (DEFAULT_TABLE table=default.Table_7c6e82a967ab492ebf7d644969562282 [id=f61209500a4941279a01276631d3b1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.494200 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7612aae4a39a41d781a2b654b2999b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.493275 1479 tablet_service.cc:1505] Processing CreateTablet for tablet a52df993b09d43eeab7ffa9144ec6289 (DEFAULT_TABLE table=default.Table_cd3855acb8cc4a7486db5bbc7d37a26e [id=487680c59db34cb8be159611a56895b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.491576 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 7a95dbd3fdb241369e1bb4ebc72bc91c (DEFAULT_TABLE table=default.Table_3a4994ec76cc4afaa6e9bb2a0fa8a353 [id=d83692ef452047daa39a765057bb704b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.494707 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52df993b09d43eeab7ffa9144ec6289. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.498638 1766 consensus_queue.cc:260] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.499887 1766 ts_tablet_manager.cc:1434] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20251024 19:48:24.500736 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a95dbd3fdb241369e1bb4ebc72bc91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.500818 1766 tablet_bootstrap.cc:492] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:24.503010 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 7a95dbd3fdb241369e1bb4ebc72bc91c (DEFAULT_TABLE table=default.Table_3a4994ec76cc4afaa6e9bb2a0fa8a353 [id=d83692ef452047daa39a765057bb704b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.503134 1352 tablet_service.cc:1505] Processing CreateTablet for tablet aa55ecf90e3141fba598df64343c0f6e (DEFAULT_TABLE table=default.Table_9108635a402d40dba89df9a92ae028b7 [id=8c070d1f7a454c6aba743f27624c92c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.503371 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a95dbd3fdb241369e1bb4ebc72bc91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.505043 1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa55ecf90e3141fba598df64343c0f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.505703 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 7612aae4a39a41d781a2b654b2999b32 (DEFAULT_TABLE table=default.Table_7c6e82a967ab492ebf7d644969562282 [id=f61209500a4941279a01276631d3b1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.506124 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7612aae4a39a41d781a2b654b2999b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.506989 1342 tablet_service.cc:1505] Processing CreateTablet for tablet a52df993b09d43eeab7ffa9144ec6289 (DEFAULT_TABLE table=default.Table_cd3855acb8cc4a7486db5bbc7d37a26e [id=487680c59db34cb8be159611a56895b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.507664 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52df993b09d43eeab7ffa9144ec6289. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.508262 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 02e6ab4d86954cb6b00376cc8d52e3c6 (DEFAULT_TABLE table=default.Table_9fcc7f42f92641fdbef94211601c4a32 [id=7519e13af9084a5d8ff34dc84c27f4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.508731 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e6ab4d86954cb6b00376cc8d52e3c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.508668 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 4d4069cbbee04c15997a6935cdc15f40 (DEFAULT_TABLE table=default.Table_932da5af90f847b893732f7f8e1b982d [id=f17520fd27ba4c1fbb18ffb1e9aa5c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.492707 1489 tablet_service.cc:1505] Processing CreateTablet for tablet aa55ecf90e3141fba598df64343c0f6e (DEFAULT_TABLE table=default.Table_9108635a402d40dba89df9a92ae028b7 [id=8c070d1f7a454c6aba743f27624c92c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.509009 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d4069cbbee04c15997a6935cdc15f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.509172 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 7b503fe19ecc44ee8086855d748b36aa (DEFAULT_TABLE table=default.Table_155f8d10b0d84715a1308d35a0d2b33a [id=be649bc161114b59a47492ef5afd3bb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.509716 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 02e6ab4d86954cb6b00376cc8d52e3c6 (DEFAULT_TABLE table=default.Table_9fcc7f42f92641fdbef94211601c4a32 [id=7519e13af9084a5d8ff34dc84c27f4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.515677 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 8b445098bfd04729808641fcfb76927d (DEFAULT_TABLE table=default.Table_69b9df2416fe4680b34e9b399ef39216 [id=34414c03d49a4450a40daa6f4ea7f7de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.516085 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b445098bfd04729808641fcfb76927d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.516620 1613 tablet_service.cc:1505] Processing CreateTablet for tablet fe8656f221984776ad3978b9e651654c (DEFAULT_TABLE table=default.Table_a65cc71737804784befd07c58aefdd0b [id=d124bda7ea4c4d17918021fdfc8e8662]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.517019 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe8656f221984776ad3978b9e651654c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.521394 1339 tablet_service.cc:1505] Processing CreateTablet for tablet f73069b9201449d08a4f7b246444e891 (DEFAULT_TABLE table=default.Table_3b59a05b195a4d728ae60c0e25c5037f [id=3d9bfe7a3a90415980fc639c42d38e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.521785 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73069b9201449d08a4f7b246444e891. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:24.522241 1177 consensus_peers.cc:597] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c -> Peer 82e82847f8904b59b03f446b7fce6233 (127.0.252.253:41375): Couldn't send request to peer 82e82847f8904b59b03f446b7fce6233. Status: Network error: Client connection negotiation failed: client connection to 127.0.252.253:41375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:24.524039 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8eec3aa8f33746c09b63c8f41168eab1 consensus state has no leader
I20251024 19:48:24.525307 1766 tablet_bootstrap.cc:654] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.468847 1767 tablet_bootstrap.cc:492] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:24.528513 1767 tablet_bootstrap.cc:654] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.529109 1335 tablet_service.cc:1505] Processing CreateTablet for tablet fe8656f221984776ad3978b9e651654c (DEFAULT_TABLE table=default.Table_a65cc71737804784befd07c58aefdd0b [id=d124bda7ea4c4d17918021fdfc8e8662]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.529462 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe8656f221984776ad3978b9e651654c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.532387 1767 tablet_bootstrap.cc:492] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:24.532898 1482 tablet_service.cc:1505] Processing CreateTablet for tablet fe8656f221984776ad3978b9e651654c (DEFAULT_TABLE table=default.Table_a65cc71737804784befd07c58aefdd0b [id=d124bda7ea4c4d17918021fdfc8e8662]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.533303 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 4d4069cbbee04c15997a6935cdc15f40 (DEFAULT_TABLE table=default.Table_932da5af90f847b893732f7f8e1b982d [id=f17520fd27ba4c1fbb18ffb1e9aa5c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.533650 1767 ts_tablet_manager.cc:1403] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.000s
I20251024 19:48:24.534385 1767 raft_consensus.cc:359] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.535112 1474 tablet_service.cc:1505] Processing CreateTablet for tablet f73069b9201449d08a4f7b246444e891 (DEFAULT_TABLE table=default.Table_3b59a05b195a4d728ae60c0e25c5037f [id=3d9bfe7a3a90415980fc639c42d38e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.535699 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 4d4069cbbee04c15997a6935cdc15f40 (DEFAULT_TABLE table=default.Table_932da5af90f847b893732f7f8e1b982d [id=f17520fd27ba4c1fbb18ffb1e9aa5c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.535797 1767 raft_consensus.cc:385] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.535866 1767 raft_consensus.cc:740] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.536036 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 8b445098bfd04729808641fcfb76927d (DEFAULT_TABLE table=default.Table_69b9df2416fe4680b34e9b399ef39216 [id=34414c03d49a4450a40daa6f4ea7f7de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.536818 1766 tablet_bootstrap.cc:492] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:24.536940 1766 ts_tablet_manager.cc:1403] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20251024 19:48:24.537679 1766 raft_consensus.cc:359] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.537843 1766 raft_consensus.cc:385] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.537901 1766 raft_consensus.cc:740] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.538122 1766 consensus_queue.cc:260] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.543408 1766 ts_tablet_manager.cc:1434] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251024 19:48:24.543689 1767 consensus_queue.cc:260] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.545326 1767 ts_tablet_manager.cc:1434] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20251024 19:48:24.548331 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d4069cbbee04c15997a6935cdc15f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.548625 1767 tablet_bootstrap.cc:492] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:24.553699 1767 tablet_bootstrap.cc:654] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:24.561787 1586 consensus_peers.cc:597] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:24.569206 1767 tablet_bootstrap.cc:492] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:24.569474 1767 ts_tablet_manager.cc:1403] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.001s
I20251024 19:48:24.570361 1767 raft_consensus.cc:359] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:24.570755 1767 raft_consensus.cc:385] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.570894 1767 raft_consensus.cc:740] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.571177 1767 consensus_queue.cc:260] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:24.571825 1767 ts_tablet_manager.cc:1434] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:24.573014 1767 tablet_bootstrap.cc:492] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:24.573493 2858 raft_consensus.cc:493] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.573660 2858 raft_consensus.cc:515] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.574272 2858 leader_election.cc:290] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:24.574909 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0d5d210d8d8499cbfdbdaa29e5bb40b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:24.575588 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0d5d210d8d8499cbfdbdaa29e5bb40b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:24.576256 1444 leader_election.cc:343] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.578061 1774 ts_tablet_manager.cc:1929] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:24.578380 1767 tablet_bootstrap.cc:654] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:24.582150 1447 leader_election.cc:343] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.582278 1447 leader_election.cc:304] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:24.582551 2858 raft_consensus.cc:2749] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:24.587944 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4b4904080b7a449fb92e773032808533
I20251024 19:48:24.588424 1183 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51416 (request call id 1358) took 1001 ms. Trace:
I20251024 19:48:24.588527 1183 rpcz_store.cc:276] 1024 19:48:23.587147 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.588267 (+ 1120us) service_pool.cc:225] Handling call
1024 19:48:23.589011 (+ 744us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1024 19:48:23.589174 (+ 163us) catalog_manager.cc:2815] Modifying in-memory table state
1024 19:48:23.589286 (+ 112us) catalog_manager.cc:2824] Locking tablets
1024 19:48:23.951356 (+362070us) catalog_manager.cc:2838] Removing table and tablets from system table
1024 19:48:24.587772 (+636416us) catalog_manager.cc:2856] Removing table from by-name map
1024 19:48:24.587848 (+ 76us) catalog_manager.cc:2872] Committing in-memory state
1024 19:48:24.587884 (+ 36us) catalog_manager.cc:2876] Aborting table tasks
1024 19:48:24.588406 (+ 522us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1024 19:48:24.229720 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1024 19:48:24.229830 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
1024 19:48:24.229835 (+ 5us) write_op.cc:435] Acquired schema lock
1024 19:48:24.229838 (+ 3us) tablet.cc:660] Decoding operations
1024 19:48:24.229946 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
1024 19:48:24.229956 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1024 19:48:24.229960 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1024 19:48:24.230052 (+ 92us) tablet.cc:699] Row locks acquired
1024 19:48:24.230054 (+ 2us) write_op.cc:260] PREPARE: finished
1024 19:48:24.230087 (+ 33us) write_op.cc:270] Start()
1024 19:48:24.298944 (+ 68857us) write_op.cc:276] Timestamp: P: 1761335304230030 usec, L: 0
1024 19:48:24.298951 (+ 7us) op_driver.cc:348] REPLICATION: starting
1024 19:48:24.299198 (+ 247us) log.cc:844] Serialized 731 byte log entry
1024 19:48:24.587131 (+287933us) op_driver.cc:464] REPLICATION: finished
1024 19:48:24.587220 (+ 89us) write_op.cc:301] APPLY: starting
1024 19:48:24.587239 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1024 19:48:24.587287 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1024 19:48:24.587314 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1024 19:48:24.587386 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
1024 19:48:24.587422 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1024 19:48:24.587428 (+ 6us) write_op.cc:312] APPLY: finished
1024 19:48:24.587539 (+ 111us) log.cc:844] Serialized 29 byte log entry
1024 19:48:24.587603 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:24.587625 (+ 22us) write_op.cc:454] Released schema lock
1024 19:48:24.587632 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":36,"apply.run_cpu_time_us":473,"apply.run_wall_time_us":473,"num_ops":2,"prepare.queue_time_us":277590,"prepare.run_cpu_time_us":880,"prepare.run_wall_time_us":69589,"replication_time_us":288146}]]}
I20251024 19:48:24.591771 1767 tablet_bootstrap.cc:492] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:24.591986 1767 ts_tablet_manager.cc:1403] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20251024 19:48:24.592865 1767 raft_consensus.cc:359] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.593004 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 91bb076b4b0d4b04955043248460d04c
I20251024 19:48:24.593124 1767 raft_consensus.cc:385] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.593235 1767 raft_consensus.cc:740] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.593451 1192 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51416 (request call id 1347) took 1107 ms. Trace:
I20251024 19:48:24.593566 1767 consensus_queue.cc:260] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.593531 1192 rpcz_store.cc:276] 1024 19:48:23.485864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.485925 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:23.486671 (+ 746us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1024 19:48:23.486838 (+ 167us) catalog_manager.cc:2815] Modifying in-memory table state
1024 19:48:23.486906 (+ 68us) catalog_manager.cc:2824] Locking tablets
1024 19:48:23.979192 (+492286us) catalog_manager.cc:2838] Removing table and tablets from system table
1024 19:48:24.592841 (+613649us) catalog_manager.cc:2856] Removing table from by-name map
1024 19:48:24.592925 (+ 84us) catalog_manager.cc:2872] Committing in-memory state
1024 19:48:24.592976 (+ 51us) catalog_manager.cc:2876] Aborting table tasks
1024 19:48:24.593434 (+ 458us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1024 19:48:24.299337 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1024 19:48:24.337882 (+ 38545us) write_op.cc:432] Acquiring schema lock in shared mode
1024 19:48:24.337891 (+ 9us) write_op.cc:435] Acquired schema lock
1024 19:48:24.337893 (+ 2us) tablet.cc:660] Decoding operations
1024 19:48:24.338034 (+ 141us) write_op.cc:620] Acquiring the partition lock for write op
1024 19:48:24.338047 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1024 19:48:24.338053 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1024 19:48:24.338195 (+ 142us) tablet.cc:699] Row locks acquired
1024 19:48:24.338200 (+ 5us) write_op.cc:260] PREPARE: finished
1024 19:48:24.338251 (+ 51us) write_op.cc:270] Start()
1024 19:48:24.338300 (+ 49us) write_op.cc:276] Timestamp: P: 1761335304338226 usec, L: 0
1024 19:48:24.338306 (+ 6us) op_driver.cc:348] REPLICATION: starting
1024 19:48:24.339659 (+ 1353us) log.cc:844] Serialized 762 byte log entry
1024 19:48:24.592036 (+252377us) op_driver.cc:464] REPLICATION: finished
1024 19:48:24.592419 (+ 383us) write_op.cc:301] APPLY: starting
1024 19:48:24.592432 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1024 19:48:24.592466 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
1024 19:48:24.592486 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
1024 19:48:24.592549 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
1024 19:48:24.592578 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1024 19:48:24.592585 (+ 7us) write_op.cc:312] APPLY: finished
1024 19:48:24.592648 (+ 63us) log.cc:844] Serialized 29 byte log entry
1024 19:48:24.592699 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:24.592715 (+ 16us) write_op.cc:454] Released schema lock
1024 19:48:24.592720 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":335,"apply.run_cpu_time_us":367,"apply.run_wall_time_us":366,"num_ops":2,"prepare.queue_time_us":319350,"prepare.run_cpu_time_us":1026,"prepare.run_wall_time_us":40411,"replication_time_us":253704}]]}
I20251024 19:48:24.594416 1767 ts_tablet_manager.cc:1434] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:24.595314 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 91bb076b4b0d4b04955043248460d04c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:24.595721 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet 91bb076b4b0d4b04955043248460d04c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.596150 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b4904080b7a449fb92e773032808533 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.596782 1767 tablet_bootstrap.cc:492] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:24.597386 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 9229fa6bf13441fa894e7bb7f78d7038
I20251024 19:48:24.597800 1191 catalog_manager.cc:6719] ReplaceTablet: tablet 9229fa6bf13441fa894e7bb7f78d7038 deleted and replaced by tablet 83bb9b2dbf5c48f38ed8b1f29d1ba466
I20251024 19:48:24.597890 1191 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51416 (request call id 1349) took 1103 ms. Trace:
I20251024 19:48:24.597956 1191 rpcz_store.cc:276] 1024 19:48:23.494406 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.494470 (+ 64us) service_pool.cc:225] Handling call
1024 19:48:24.597878 (+1103408us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1024 19:48:24.339798 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1024 19:48:24.339889 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
1024 19:48:24.339895 (+ 6us) write_op.cc:435] Acquired schema lock
1024 19:48:24.339897 (+ 2us) tablet.cc:660] Decoding operations
1024 19:48:24.340004 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
1024 19:48:24.340014 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1024 19:48:24.340020 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1024 19:48:24.340096 (+ 76us) tablet.cc:699] Row locks acquired
1024 19:48:24.340098 (+ 2us) write_op.cc:260] PREPARE: finished
1024 19:48:24.340126 (+ 28us) write_op.cc:270] Start()
1024 19:48:24.340157 (+ 31us) write_op.cc:276] Timestamp: P: 1761335304340109 usec, L: 0
1024 19:48:24.340161 (+ 4us) op_driver.cc:348] REPLICATION: starting
1024 19:48:24.340435 (+ 274us) log.cc:844] Serialized 628 byte log entry
1024 19:48:24.595639 (+255204us) op_driver.cc:464] REPLICATION: finished
1024 19:48:24.595714 (+ 75us) write_op.cc:301] APPLY: starting
1024 19:48:24.595725 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1024 19:48:24.595767 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1024 19:48:24.595788 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
1024 19:48:24.595940 (+ 152us) tablet.cc:1381] finished ApplyRowOperation cycle
1024 19:48:24.596008 (+ 68us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1024 19:48:24.596015 (+ 7us) write_op.cc:312] APPLY: finished
1024 19:48:24.596082 (+ 67us) log.cc:844] Serialized 29 byte log entry
1024 19:48:24.596133 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
1024 19:48:24.596148 (+ 15us) write_op.cc:454] Released schema lock
1024 19:48:24.596153 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":35,"apply.run_cpu_time_us":490,"apply.run_wall_time_us":490,"num_ops":2,"prepare.queue_time_us":359073,"prepare.run_cpu_time_us":646,"prepare.run_wall_time_us":717,"replication_time_us":255454,"spinlock_wait_cycles":768}]]}
I20251024 19:48:24.598910 1767 tablet_bootstrap.cc:654] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.599542 1510 raft_consensus.cc:1275] T a1fbbba88a294adfbf97c0343d86698e P 10a9536207e343f188e0ef709081d04e [term 2 FOLLOWER]: Refusing update from remote peer 43bec2f03fc849099e254f6335e9d532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251024 19:48:24.599718 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 9229fa6bf13441fa894e7bb7f78d7038 with delete_type TABLET_DATA_DELETED (replaced by tablet 83bb9b2dbf5c48f38ed8b1f29d1ba466) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:24.600297 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 9229fa6bf13441fa894e7bb7f78d7038 with delete_type TABLET_DATA_DELETED (replaced by tablet 83bb9b2dbf5c48f38ed8b1f29d1ba466) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:24.600984 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83bb9b2dbf5c48f38ed8b1f29d1ba466 has no consensus state
I20251024 19:48:24.601325 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e6ab4d86954cb6b00376cc8d52e3c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.604383 1185 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (table default.Table_18e7285b7df44399ab3be1dfc86eb91f [id=14bda7f9324f44aa9cb87a5f129039be]). Expected version 2 got 0
I20251024 19:48:24.606231 1184 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (table default.Table_18e7285b7df44399ab3be1dfc86eb91f [id=14bda7f9324f44aa9cb87a5f129039be]). Expected version 2 got 0
I20251024 19:48:24.606587 2923 consensus_queue.cc:1048] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:24.608711 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 83b91a5d764449d5b82ef6d1ee0360f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:24.608886 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a1031d785b646b095ece9be666d2a77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.610512 1189 master_service.cc:954] ReplaceTablet: received request to replace tablet 95e11ed28aab4e809e8fc1615f21744c from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:24.611155 1767 tablet_bootstrap.cc:492] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:24.611621 1767 ts_tablet_manager.cc:1403] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.001s
I20251024 19:48:24.612931 1767 raft_consensus.cc:359] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.613392 1767 raft_consensus.cc:385] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.613530 1767 raft_consensus.cc:740] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.613668 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet 913fee0ca74e4cf99d6697ad5c38f4eb from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:24.613907 1767 consensus_queue.cc:260] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.615780 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_98de64dffcd34afc91c487faa22cb705" } new_table_name: "default.Table_21c23f21b4f24c9989cdfe18a12a4693" modify_external_catalogs: true
I20251024 19:48:24.616564 1766 tablet_bootstrap.cc:492] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:24.616770 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 250f106664be4b5aaa330dbb4e43cb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.618244 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_babcfed7fd5b4b6ebf17c81871843326" } modify_external_catalogs: true
I20251024 19:48:24.618382 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_babcfed7fd5b4b6ebf17c81871843326" } modify_external_catalogs: true
I20251024 19:48:24.618499 1766 tablet_bootstrap.cc:654] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.618963 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0929cd0dc2cb455ca05d93504fb52ca8" } new_table_name: "default.Table_6428726d0a954e8fb87d61c3b41f1bc8" modify_external_catalogs: true
I20251024 19:48:24.619553 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d4069cbbee04c15997a6935cdc15f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.619745 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe8656f221984776ad3978b9e651654c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.620491 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_bd503f203f9943b19088dca94ce8af98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:24.621114 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd503f203f9943b19088dca94ce8af98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:24.621696 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b503fe19ecc44ee8086855d748b36aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.622638 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c0b98ba82d324740b9152d554bea55d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:24.623298 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0b98ba82d324740b9152d554bea55d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:24.624348 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_50a7c4f2e9b94c2f9781f366cbac5b92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:24.624734 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50a7c4f2e9b94c2f9781f366cbac5b92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:24.624797 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa55ecf90e3141fba598df64343c0f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.628974 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2c4240fa3c4d4fa5bd42eeb197a2088d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:24.629604 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c4240fa3c4d4fa5bd42eeb197a2088d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:24.631498 1767 ts_tablet_manager.cc:1434] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20251024 19:48:24.632119 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_54c5eb4772114f2e829fe31c91a9292d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:24.632786 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54c5eb4772114f2e829fe31c91a9292d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:24.634361 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b4904080b7a449fb92e773032808533 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:24.636086 1766 tablet_bootstrap.cc:492] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:24.636298 1766 ts_tablet_manager.cc:1403] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.000s
I20251024 19:48:24.636402 1774 log.cc:1199] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/9026d9e801294dd2bfdb9c90e45a0990
I20251024 19:48:24.637037 1766 raft_consensus.cc:359] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.637203 1766 raft_consensus.cc:385] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.637252 1766 raft_consensus.cc:740] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.637447 1766 consensus_queue.cc:260] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.638152 1766 ts_tablet_manager.cc:1434] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:24.638465 1766 tablet_bootstrap.cc:492] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:24.639418 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 043fac06e6324032a61c42a10d51ea22 consensus state has no leader
I20251024 19:48:24.639904 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73069b9201449d08a4f7b246444e891. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.639918 1766 tablet_bootstrap.cc:654] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.640805 1768 tablet_bootstrap.cc:492] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:24.640944 1768 ts_tablet_manager.cc:1403] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.283s user 0.005s sys 0.002s
I20251024 19:48:24.641436 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 95e11ed28aab4e809e8fc1615f21744c
I20251024 19:48:24.641911 1189 catalog_manager.cc:6719] ReplaceTablet: tablet 95e11ed28aab4e809e8fc1615f21744c deleted and replaced by tablet c215211de857468d9c28a9a87b29ef4f
I20251024 19:48:24.645394 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 4641256cffe945b58ef01eecf402d4c6 (DEFAULT_TABLE table=default.Table_e517c79e1029452c8529bc3aef0c8338 [id=c982a95269464e1ca9d2bcec333939de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.645452 1338 tablet_service.cc:1505] Processing CreateTablet for tablet e2050cb06f5b47d4adca131432d9103d (DEFAULT_TABLE table=default.Table_260350863f8e488f85b754a90b8facdd [id=7ad729e53c024b3e8bff7222f5f1f624]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.645841 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4641256cffe945b58ef01eecf402d4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.647818 1766 tablet_bootstrap.cc:492] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:24.647800 1476 tablet_service.cc:1505] Processing CreateTablet for tablet e2050cb06f5b47d4adca131432d9103d (DEFAULT_TABLE table=default.Table_260350863f8e488f85b754a90b8facdd [id=7ad729e53c024b3e8bff7222f5f1f624]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.647990 1766 ts_tablet_manager.cc:1403] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20251024 19:48:24.648178 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2050cb06f5b47d4adca131432d9103d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.648695 1766 raft_consensus.cc:359] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.648855 1766 raft_consensus.cc:385] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.648905 1766 raft_consensus.cc:740] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.649116 1766 consensus_queue.cc:260] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.649632 1766 ts_tablet_manager.cc:1434] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:24.649892 1766 tablet_bootstrap.cc:492] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:24.651333 1766 tablet_bootstrap.cc:654] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.652170 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 4641256cffe945b58ef01eecf402d4c6 (DEFAULT_TABLE table=default.Table_e517c79e1029452c8529bc3aef0c8338 [id=c982a95269464e1ca9d2bcec333939de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.653128 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1017) took 4568 ms. Trace:
I20251024 19:48:24.653290 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 4641256cffe945b58ef01eecf402d4c6 (DEFAULT_TABLE table=default.Table_e517c79e1029452c8529bc3aef0c8338 [id=c982a95269464e1ca9d2bcec333939de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.653240 1775 rpcz_store.cc:276] 1024 19:48:20.085038 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.085090 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:24.253199 (+4168109us) ts_tablet_manager.cc:1151] Deleting tablet 65424b00ab9c454f9584048caba83e6a
1024 19:48:24.253208 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.334968 (+ 81760us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.479005 (+144037us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.653110 (+174105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4167967}
I20251024 19:48:24.653620 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 8b445098bfd04729808641fcfb76927d (DEFAULT_TABLE table=default.Table_69b9df2416fe4680b34e9b399ef39216 [id=34414c03d49a4450a40daa6f4ea7f7de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.653813 1627 tablet_service.cc:1505] Processing CreateTablet for tablet 88f6f9259fef4eb3b329e4ebbfe822e4 (DEFAULT_TABLE table=default.Table_737f344a623f459aa91cedb039b78ffe [id=ffc1b30bc760477696e21d255359b7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.654009 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 65424b00ab9c454f9584048caba83e6a (table default.Table_e6ec362938cc478d9b9abee36f34f5c5 [id=d2e2211a2cd8457aab95305acead0b3b]) successfully deleted
I20251024 19:48:24.654142 1611 tablet_service.cc:1505] Processing CreateTablet for tablet e2050cb06f5b47d4adca131432d9103d (DEFAULT_TABLE table=default.Table_260350863f8e488f85b754a90b8facdd [id=7ad729e53c024b3e8bff7222f5f1f624]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.654477 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b445098bfd04729808641fcfb76927d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.654618 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1018) took 4556 ms. Trace:
I20251024 19:48:24.654767 1775 rpcz_store.cc:276] 1024 19:48:20.098288 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.098351 (+ 63us) service_pool.cc:225] Handling call
1024 19:48:24.654325 (+4555974us) ts_tablet_manager.cc:1151] Deleting tablet 8d1b6eaa353c4284a2b114b42437272c
1024 19:48:24.654343 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.654589 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4555752}
W20251024 19:48:24.655467 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 8d1b6eaa353c4284a2b114b42437272c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1b6eaa353c4284a2b114b42437272c already in progress: creating tablet
I20251024 19:48:24.656055 1472 tablet_service.cc:1552] Processing DeleteTablet for tablet 95e11ed28aab4e809e8fc1615f21744c with delete_type TABLET_DATA_DELETED (replaced by tablet c215211de857468d9c28a9a87b29ef4f) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:24.658233 1767 tablet_bootstrap.cc:492] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:24.659248 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 95e11ed28aab4e809e8fc1615f21744c with delete_type TABLET_DATA_DELETED (replaced by tablet c215211de857468d9c28a9a87b29ef4f) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.659925 1767 tablet_bootstrap.cc:654] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.660166 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 88f6f9259fef4eb3b329e4ebbfe822e4 (DEFAULT_TABLE table=default.Table_737f344a623f459aa91cedb039b78ffe [id=ffc1b30bc760477696e21d255359b7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.660544 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f6f9259fef4eb3b329e4ebbfe822e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.660974 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet 95e11ed28aab4e809e8fc1615f21744c with delete_type TABLET_DATA_DELETED (replaced by tablet c215211de857468d9c28a9a87b29ef4f) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:24.661055 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2050cb06f5b47d4adca131432d9103d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.662182 1472 tablet_service.cc:1505] Processing CreateTablet for tablet 88f6f9259fef4eb3b329e4ebbfe822e4 (DEFAULT_TABLE table=default.Table_737f344a623f459aa91cedb039b78ffe [id=ffc1b30bc760477696e21d255359b7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.662416 1768 raft_consensus.cc:359] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.662611 1768 raft_consensus.cc:385] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.662683 1768 raft_consensus.cc:740] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.662976 1768 consensus_queue.cc:260] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.663542 1774 ts_tablet_manager.cc:1950] T 9026d9e801294dd2bfdb9c90e45a0990 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:24.664762 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1028) took 1536 ms. Trace:
I20251024 19:48:24.665026 2858 raft_consensus.cc:493] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.665215 2858 raft_consensus.cc:515] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.665694 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4641256cffe945b58ef01eecf402d4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.665622 1774 rpcz_store.cc:276] 1024 19:48:23.127997 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.133699 (+ 5702us) service_pool.cc:225] Handling call
1024 19:48:24.382165 (+1248466us) ts_tablet_manager.cc:1151] Deleting tablet 9026d9e801294dd2bfdb9c90e45a0990
1024 19:48:24.382174 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.547167 (+164993us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.577957 (+ 30790us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.664734 (+ 86777us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1248231}
I20251024 19:48:24.666055 2858 leader_election.cc:290] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:24.666100 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 9026d9e801294dd2bfdb9c90e45a0990 (table default.Table_9e14814a2db64e0b9602e41f926f3b62 [id=1841054cd03e4989a236ebf134fa7755]) successfully deleted
I20251024 19:48:24.666749 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1034) took 1482 ms. Trace:
I20251024 19:48:24.666723 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e7b3288182744969e217d4f174fe4ea" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:24.666817 1774 rpcz_store.cc:276] 1024 19:48:23.184765 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.184812 (+ 47us) service_pool.cc:225] Handling call
1024 19:48:24.666615 (+1481803us) ts_tablet_manager.cc:1151] Deleting tablet cb1f850b3e104707a53d540ad5752d4d
1024 19:48:24.666625 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.666732 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1471621}
I20251024 19:48:24.666945 1642 raft_consensus.cc:2468] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
W20251024 19:48:24.667169 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet cb1f850b3e104707a53d540ad5752d4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb1f850b3e104707a53d540ad5752d4d already in progress: creating tablet
I20251024 19:48:24.667398 1444 leader_election.cc:304] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:24.669811 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f6f9259fef4eb3b329e4ebbfe822e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.671036 2858 raft_consensus.cc:2804] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:24.671118 2858 raft_consensus.cc:493] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:24.671178 2858 raft_consensus.cc:3060] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:24.671794 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2050cb06f5b47d4adca131432d9103d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.673866 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e7b3288182744969e217d4f174fe4ea" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:24.674458 1447 leader_election.cc:343] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.674564 2858 raft_consensus.cc:515] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.675732 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e7b3288182744969e217d4f174fe4ea" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:24.675904 1642 raft_consensus.cc:3060] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:24.678720 1642 raft_consensus.cc:2468] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:24.679237 1444 leader_election.cc:304] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:24.679523 3390 raft_consensus.cc:2804] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:24.684063 1768 ts_tablet_manager.cc:1434] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20251024 19:48:24.684432 1768 tablet_bootstrap.cc:492] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:24.685707 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e7b3288182744969e217d4f174fe4ea" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:24.686305 1447 leader_election.cc:343] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.686972 2858 leader_election.cc:290] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:24.690459 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1035) took 1503 ms. Trace:
I20251024 19:48:24.691349 1774 rpcz_store.cc:276] 1024 19:48:23.186818 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.188956 (+ 2138us) service_pool.cc:225] Handling call
1024 19:48:24.690333 (+1501377us) ts_tablet_manager.cc:1151] Deleting tablet b24a3a820c9b496dbf6955b399c6ea8c
1024 19:48:24.690343 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.690438 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1483848}
I20251024 19:48:24.691979 1776 ts_tablet_manager.cc:1929] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:24.692540 1766 tablet_bootstrap.cc:492] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:24.693404 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1046) took 1227 ms. Trace:
W20251024 19:48:24.693465 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b24a3a820c9b496dbf6955b399c6ea8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b24a3a820c9b496dbf6955b399c6ea8c already in progress: creating tablet
I20251024 19:48:24.693562 1774 rpcz_store.cc:276] 1024 19:48:23.465695 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.465785 (+ 90us) service_pool.cc:225] Handling call
1024 19:48:24.693285 (+1227500us) ts_tablet_manager.cc:1151] Deleting tablet 830c8469331940fc85aaad58c9085e35
1024 19:48:24.693297 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.693387 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1227147}
W20251024 19:48:24.694113 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 830c8469331940fc85aaad58c9085e35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 830c8469331940fc85aaad58c9085e35 already in progress: creating tablet
I20251024 19:48:24.695058 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1047) took 1228 ms. Trace:
I20251024 19:48:24.695178 1766 ts_tablet_manager.cc:1403] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.009s
I20251024 19:48:24.695312 1774 rpcz_store.cc:276] 1024 19:48:23.466577 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.466658 (+ 81us) service_pool.cc:225] Handling call
1024 19:48:24.694419 (+1227761us) ts_tablet_manager.cc:1151] Deleting tablet 4846799df5b242ce8f19c034320c8a1d
1024 19:48:24.694429 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.694513 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1227592}
I20251024 19:48:24.696020 1766 raft_consensus.cc:359] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.696195 1766 raft_consensus.cc:385] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.696244 1766 raft_consensus.cc:740] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.696429 1766 consensus_queue.cc:260] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:24.696597 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4846799df5b242ce8f19c034320c8a1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4846799df5b242ce8f19c034320c8a1d already in progress: creating tablet
I20251024 19:48:24.696785 1774 tablet_replica.cc:333] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:24.697033 1774 raft_consensus.cc:2243] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:24.697041 1766 ts_tablet_manager.cc:1434] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:24.697352 1766 tablet_bootstrap.cc:492] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:24.697683 1776 log.cc:1199] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/1edc55caab904722bf19a1c71df601cf
I20251024 19:48:24.698251 1776 ts_tablet_manager.cc:1950] T 1edc55caab904722bf19a1c71df601cf P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:24.699456 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1053) took 3499 ms. Trace:
I20251024 19:48:24.700722 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 0f2df2b286024ae8be02b29041c36803 (DEFAULT_TABLE table=default.Table_fc6e97f574374fe4972a47db87f70001 [id=07ca9611d29d40eda270ab85b8457200]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.703949 1776 rpcz_store.cc:276] 1024 19:48:21.200297 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.200765 (+ 468us) service_pool.cc:225] Handling call
1024 19:48:24.309295 (+3108530us) ts_tablet_manager.cc:1151] Deleting tablet 1edc55caab904722bf19a1c71df601cf
1024 19:48:24.309304 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.586377 (+277073us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.691849 (+105472us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.699437 (+ 7588us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3108370}
I20251024 19:48:24.656352 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4641256cffe945b58ef01eecf402d4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.704802 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f6f9259fef4eb3b329e4ebbfe822e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.705416 1776 tablet_replica.cc:333] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:24.705833 1776 raft_consensus.cc:2243] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:24.706432 1776 raft_consensus.cc:2272] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:24.708782 1776 ts_tablet_manager.cc:1916] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:24.712175 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 1edc55caab904722bf19a1c71df601cf (table default.Table_641c979b063f419884a09e448c68a1b4 [id=b72bff5d78ec4eb08810442a084a16a8]) successfully deleted
I20251024 19:48:24.713167 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f2df2b286024ae8be02b29041c36803. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.715638 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 0f2df2b286024ae8be02b29041c36803 (DEFAULT_TABLE table=default.Table_fc6e97f574374fe4972a47db87f70001 [id=07ca9611d29d40eda270ab85b8457200]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.716410 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_069227ac82b5485b9f88980affdf85ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:24.716878 1766 tablet_bootstrap.cc:654] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:24.716956 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_069227ac82b5485b9f88980affdf85ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:24.719921 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (table default.Table_18e7285b7df44399ab3be1dfc86eb91f [id=14bda7f9324f44aa9cb87a5f129039be]). Expected version 2 got 0
I20251024 19:48:24.720404 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet a1fbbba88a294adfbf97c0343d86698e (table default.Table_7bc8cd4a930b44dfafa473297875594d [id=a60c8222aa0248c091fab8dc34100847]). Expected version 2 got 0
I20251024 19:48:24.720593 1766 tablet_bootstrap.cc:492] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:24.720732 1766 ts_tablet_manager.cc:1403] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20251024 19:48:24.720850 1185 catalog_manager.cc:5649] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:24.721266 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 1999b751e9344dcf81d26226c0589c8f (table default.Table_9147a98b3bf9483ea7b7a6f17130d4fc [id=d743705c21da4b89abedf2eb34ddbe09]). Expected version 1 got 0
I20251024 19:48:24.721467 1766 raft_consensus.cc:359] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.721763 1766 raft_consensus.cc:385] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.721820 1766 raft_consensus.cc:740] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.721632 1185 catalog_manager.cc:5649] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:24.721994 1766 consensus_queue.cc:260] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.722445 1766 ts_tablet_manager.cc:1434] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:24.722412 1185 catalog_manager.cc:5649] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:24.722785 1766 tablet_bootstrap.cc:492] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:24.723024 1185 catalog_manager.cc:5649] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:24.724807 3443 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8906492b1a4368936fac95c11e1f73" member_type: VOTER last_known_addr { host: "127.0.252.254" port: 36505 } } peers { permanent_uuid: "82e82847f8904b59b03f446b7fce6233" member_type: VOTER last_known_addr { host: "127.0.252.253" port: 41375 } } peers { permanent_uuid: "dc8f71a5b70749a4a3e3a599bcb3d22c" member_type: VOTER last_known_addr { host: "127.0.252.252" port: 36627 } } }
I20251024 19:48:24.725097 3443 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c [sys.catalog]: This master's current role is: LEADER
W20251024 19:48:24.730049 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 08b1b461a5de434d83ba5b6c4587a66f consensus state has no leader
I20251024 19:48:24.734324 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f2df2b286024ae8be02b29041c36803. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.736276 1766 tablet_bootstrap.cc:654] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.741329 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2f3f9f0797ae454eb0fd2975c56cadfd" } modify_external_catalogs: true
I20251024 19:48:24.741528 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2f3f9f0797ae454eb0fd2975c56cadfd" } modify_external_catalogs: true
I20251024 19:48:24.742065 1774 raft_consensus.cc:2272] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:24.744453 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet 3c708051e9754fe8afc45154b0a4f30c from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:24.744647 1766 tablet_bootstrap.cc:492] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:24.744781 1766 ts_tablet_manager.cc:1403] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20251024 19:48:24.599560 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 9229fa6bf13441fa894e7bb7f78d7038 with delete_type TABLET_DATA_DELETED (replaced by tablet 83bb9b2dbf5c48f38ed8b1f29d1ba466) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:24.600174 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 91bb076b4b0d4b04955043248460d04c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:24.697877 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 0f2df2b286024ae8be02b29041c36803 (DEFAULT_TABLE table=default.Table_fc6e97f574374fe4972a47db87f70001 [id=07ca9611d29d40eda270ab85b8457200]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.746246 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f2df2b286024ae8be02b29041c36803. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.746345 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_088895c10fa249e488c6cd194ea431fd" } new_table_name: "default.Table_88c9fef9aae24e17a13815ad941e02d7" modify_external_catalogs: true
I20251024 19:48:24.747812 1766 raft_consensus.cc:359] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.748445 1766 raft_consensus.cc:385] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.748689 1766 raft_consensus.cc:740] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.600905 3441 consensus_queue.cc:1048] T a1fbbba88a294adfbf97c0343d86698e P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.102s
I20251024 19:48:24.749464 1766 consensus_queue.cc:260] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.729451 1768 tablet_bootstrap.cc:654] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.591727 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b4904080b7a449fb92e773032808533 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:23 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:24.751619 1774 ts_tablet_manager.cc:1916] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:24.755019 3390 raft_consensus.cc:697] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:24.755322 1776 ts_tablet_manager.cc:1929] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:24.755437 1776 log.cc:1199] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/d840888183ec4ef7bb245363c5bac322
I20251024 19:48:24.755388 3390 consensus_queue.cc:237] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.755789 1776 ts_tablet_manager.cc:1950] T d840888183ec4ef7bb245363c5bac322 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:24.756557 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1052) took 3556 ms. Trace:
I20251024 19:48:24.756981 1766 ts_tablet_manager.cc:1434] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.012s user 0.002s sys 0.002s
I20251024 19:48:24.757118 3396 raft_consensus.cc:493] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.757282 1766 tablet_bootstrap.cc:492] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:24.757287 3396 raft_consensus.cc:515] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.758385 3396 leader_election.cc:290] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:24.758570 3396 raft_consensus.cc:493] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.758775 1766 tablet_bootstrap.cc:654] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.758736 3396 raft_consensus.cc:515] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.759613 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10891ff5bf0d4b93bcab9faa5f7346f1" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:24.760327 3455 raft_consensus.cc:493] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.760485 3455 raft_consensus.cc:515] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.761632 3455 leader_election.cc:290] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:24.761525 1335 tablet_service.cc:1505] Processing CreateTablet for tablet ac6c18e5c411477097831a1a3aeedcc3 (DEFAULT_TABLE table=default.Table_99a4ef9d7f6e40d1b12c6d8420ae7e76 [id=e3a374dc570f417db0d2785f848871c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.760177 1627 tablet_service.cc:1505] Processing CreateTablet for tablet ac6c18e5c411477097831a1a3aeedcc3 (DEFAULT_TABLE table=default.Table_99a4ef9d7f6e40d1b12c6d8420ae7e76 [id=e3a374dc570f417db0d2785f848871c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.761986 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac6c18e5c411477097831a1a3aeedcc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.762166 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80c58e2439dc490c824b5f0023b5434b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:24.762352 1476 tablet_service.cc:1505] Processing CreateTablet for tablet ac6c18e5c411477097831a1a3aeedcc3 (DEFAULT_TABLE table=default.Table_99a4ef9d7f6e40d1b12c6d8420ae7e76 [id=e3a374dc570f417db0d2785f848871c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.762712 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac6c18e5c411477097831a1a3aeedcc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.762941 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 7b503fe19ecc44ee8086855d748b36aa (DEFAULT_TABLE table=default.Table_155f8d10b0d84715a1308d35a0d2b33a [id=be649bc161114b59a47492ef5afd3bb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.763156 3387 raft_consensus.cc:493] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.763270 3387 raft_consensus.cc:515] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.764525 3387 leader_election.cc:290] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:24.764537 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_89402da0de5c4575b677a5955a538306" } modify_external_catalogs: true
I20251024 19:48:24.764726 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_89402da0de5c4575b677a5955a538306" } modify_external_catalogs: true
I20251024 19:48:24.765163 1011 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskc9NInd/build/asan/bin/kudu
/tmp/dist-test-taskc9NInd/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.252.253:41375
--webserver_interface=127.0.252.253
--webserver_port=43901
--builtin_ntp_servers=127.0.252.212:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251024 19:48:24.590256 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 9afc93ecc49347f196545d5358db3fe8 (DEFAULT_TABLE table=default.Table_5899d327ab3844daa60d50f0035a4751 [id=65e76e5db67546ddb669e3d193ee541b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:24.765861 1447 leader_election.cc:343] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.769783 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da7d38b63708427982591da1014aca81" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:24.769997 1509 raft_consensus.cc:2468] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:24.760619 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10891ff5bf0d4b93bcab9faa5f7346f1" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:24.761173 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65424b00ab9c454f9584048caba83e6a" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:24.770695 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da7d38b63708427982591da1014aca81" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:24.761400 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80c58e2439dc490c824b5f0023b5434b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:24.770836 1305 leader_election.cc:304] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:24.770440 1586 leader_election.cc:304] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:24.771212 1444 leader_election.cc:343] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.771317 1444 leader_election.cc:304] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:24.771469 3459 raft_consensus.cc:2749] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:24.774938 1586 leader_election.cc:343] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.775830 2858 raft_consensus.cc:2749] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:24.780546 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1019) took 4681 ms. Trace:
I20251024 19:48:24.780707 1775 rpcz_store.cc:276] 1024 19:48:20.099099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.099270 (+ 171us) service_pool.cc:225] Handling call
1024 19:48:24.780374 (+4681104us) ts_tablet_manager.cc:1151] Deleting tablet 3ef6d244c498452ca79cd6c6d5c003df
1024 19:48:24.780387 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.780524 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4680939}
W20251024 19:48:24.781765 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3ef6d244c498452ca79cd6c6d5c003df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ef6d244c498452ca79cd6c6d5c003df already in progress: creating tablet
I20251024 19:48:24.782034 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1020) took 4681 ms. Trace:
I20251024 19:48:24.782191 1775 rpcz_store.cc:276] 1024 19:48:20.100134 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.100184 (+ 50us) service_pool.cc:225] Handling call
1024 19:48:24.781243 (+4681059us) ts_tablet_manager.cc:1151] Deleting tablet 9287f340fb554904986f6c18b7356701
1024 19:48:24.781254 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.781380 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4680881}
I20251024 19:48:24.782878 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1021) took 4681 ms. Trace:
I20251024 19:48:24.783004 1775 rpcz_store.cc:276] 1024 19:48:20.101086 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.101141 (+ 55us) service_pool.cc:225] Handling call
1024 19:48:24.782744 (+4681603us) ts_tablet_manager.cc:1151] Deleting tablet 3c708051e9754fe8afc45154b0a4f30c
1024 19:48:24.782752 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.782861 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4681381}
I20251024 19:48:24.783221 1767 tablet_bootstrap.cc:492] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:24.783408 1767 ts_tablet_manager.cc:1403] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.125s user 0.016s sys 0.010s
I20251024 19:48:24.783631 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1022) took 4681 ms. Trace:
I20251024 19:48:24.783927 1775 rpcz_store.cc:276] 1024 19:48:20.101914 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.101967 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:24.783401 (+4681434us) ts_tablet_manager.cc:1151] Deleting tablet 3a712f52cd114e9bb07c2717e5d728fd
1024 19:48:24.783411 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.783604 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4681266}
W20251024 19:48:24.784026 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9287f340fb554904986f6c18b7356701 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9287f340fb554904986f6c18b7356701 already in progress: creating tablet
W20251024 19:48:24.784248 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3c708051e9754fe8afc45154b0a4f30c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c708051e9754fe8afc45154b0a4f30c already in progress: creating tablet
W20251024 19:48:24.784461 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3a712f52cd114e9bb07c2717e5d728fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a712f52cd114e9bb07c2717e5d728fd already in progress: creating tablet
I20251024 19:48:24.784937 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac6c18e5c411477097831a1a3aeedcc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.784902 1767 raft_consensus.cc:359] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.785188 1767 raft_consensus.cc:385] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.785287 1767 raft_consensus.cc:740] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.785596 1767 consensus_queue.cc:260] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.785948 3404 raft_consensus.cc:2804] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:24.786057 3404 raft_consensus.cc:493] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:24.786144 3404 raft_consensus.cc:3060] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:24.786324 1767 ts_tablet_manager.cc:1434] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:24.756657 1776 rpcz_store.cc:276] 1024 19:48:21.200207 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.201059 (+ 852us) service_pool.cc:225] Handling call
1024 19:48:24.705366 (+3504307us) ts_tablet_manager.cc:1151] Deleting tablet d840888183ec4ef7bb245363c5bac322
1024 19:48:24.705379 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.725329 (+ 19950us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.755219 (+ 29890us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.756536 (+ 1317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3504183}
I20251024 19:48:24.787004 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1050) took 3587 ms. Trace:
I20251024 19:48:24.787062 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet d840888183ec4ef7bb245363c5bac322 (table default.Table_cb99a44a90e849be948f4a481c07a91d [id=5edc315e5e43438f83c57c449df681a8]) successfully deleted
I20251024 19:48:24.789510 1776 rpcz_store.cc:276] 1024 19:48:21.199987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.201209 (+ 1222us) service_pool.cc:225] Handling call
1024 19:48:24.786881 (+3585672us) ts_tablet_manager.cc:1151] Deleting tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253
1024 19:48:24.786889 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.786986 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3585503}
I20251024 19:48:24.790136 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 913fee0ca74e4cf99d6697ad5c38f4eb
I20251024 19:48:24.790258 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1051) took 3590 ms. Trace:
W20251024 19:48:24.790329 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 already in progress: creating tablet
I20251024 19:48:24.790325 1776 rpcz_store.cc:276] 1024 19:48:21.200111 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.201404 (+ 1293us) service_pool.cc:225] Handling call
1024 19:48:24.790151 (+3588747us) ts_tablet_manager.cc:1151] Deleting tablet 59eca6bb1dd84c67a7e53131abaa937a
1024 19:48:24.790160 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.790242 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3588627}
I20251024 19:48:24.790596 1776 tablet_replica.cc:333] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97: stopping tablet replica
W20251024 19:48:24.790717 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 59eca6bb1dd84c67a7e53131abaa937a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59eca6bb1dd84c67a7e53131abaa937a already in progress: creating tablet
I20251024 19:48:24.790809 1776 raft_consensus.cc:2243] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:24.791152 1776 raft_consensus.cc:2272] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:24.788966 3404 raft_consensus.cc:515] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.791815 3404 leader_election.cc:290] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:24.792438 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da7d38b63708427982591da1014aca81" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:24.792621 1510 raft_consensus.cc:3060] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:24.789448 1767 tablet_bootstrap.cc:492] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:24.795946 1510 raft_consensus.cc:2468] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:24.796774 1586 leader_election.cc:304] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:24.797060 3404 raft_consensus.cc:2804] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:24.797459 3404 raft_consensus.cc:697] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:24.797895 1766 tablet_bootstrap.cc:492] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:24.797802 3404 consensus_queue.cc:237] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.798053 1766 ts_tablet_manager.cc:1403] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20251024 19:48:24.798791 1766 raft_consensus.cc:359] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.799228 1766 raft_consensus.cc:385] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.799510 1766 raft_consensus.cc:740] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.797070 1776 ts_tablet_manager.cc:1916] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:24.800156 1187 catalog_manager.cc:6719] ReplaceTablet: tablet 913fee0ca74e4cf99d6697ad5c38f4eb deleted and replaced by tablet ca38f3a765264dcc83f73a5d586cfea9
I20251024 19:48:24.802452 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b503fe19ecc44ee8086855d748b36aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.806545 1627 tablet_service.cc:1552] Processing DeleteTablet for tablet 913fee0ca74e4cf99d6697ad5c38f4eb with delete_type TABLET_DATA_DELETED (replaced by tablet ca38f3a765264dcc83f73a5d586cfea9) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.806831 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 541aebb900d6432dbf5aea81c7237120 (DEFAULT_TABLE table=default.Table_b94c179f1c064b5ab2010fe812734297 [id=b705011216694ca99040f9044380f118]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.808075 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541aebb900d6432dbf5aea81c7237120. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.808465 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9afc93ecc49347f196545d5358db3fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.808847 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 541aebb900d6432dbf5aea81c7237120 (DEFAULT_TABLE table=default.Table_b94c179f1c064b5ab2010fe812734297 [id=b705011216694ca99040f9044380f118]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.809221 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541aebb900d6432dbf5aea81c7237120. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.809522 1486 tablet_service.cc:1505] Processing CreateTablet for tablet 6491a33902bf46d093b611f82b70a5e6 (DEFAULT_TABLE table=default.Table_f3fbc16ea2444f798f1e23a68acdf61f [id=c6844395ab8c4a11801463f8a3248bdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.809875 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6491a33902bf46d093b611f82b70a5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.809845 1613 tablet_service.cc:1505] Processing CreateTablet for tablet a05a69bd7d1d4bf291d7cda76658602e (DEFAULT_TABLE table=default.Table_510bc2b3752b42b09ebef315e1e79a48 [id=d3e4be6314074414a3c9ea93b53a14d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.810180 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05a69bd7d1d4bf291d7cda76658602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.807003 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 6491a33902bf46d093b611f82b70a5e6 (DEFAULT_TABLE table=default.Table_f3fbc16ea2444f798f1e23a68acdf61f [id=c6844395ab8c4a11801463f8a3248bdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.812783 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6491a33902bf46d093b611f82b70a5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.813882 1490 tablet_service.cc:1505] Processing CreateTablet for tablet a05a69bd7d1d4bf291d7cda76658602e (DEFAULT_TABLE table=default.Table_510bc2b3752b42b09ebef315e1e79a48 [id=d3e4be6314074414a3c9ea93b53a14d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.815445 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05a69bd7d1d4bf291d7cda76658602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.815713 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 541aebb900d6432dbf5aea81c7237120 (DEFAULT_TABLE table=default.Table_b94c179f1c064b5ab2010fe812734297 [id=b705011216694ca99040f9044380f118]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:24.816445 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 8eec3aa8f33746c09b63c8f41168eab1 (table default.Table_dc0f70b1b6cd4a62b93398c8b38ef1fe [id=bd92e84f88c543918f6ad9c91b31d1f4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8eec3aa8f33746c09b63c8f41168eab1 consensus state has no leader
I20251024 19:48:24.817150 1339 tablet_service.cc:1505] Processing CreateTablet for tablet a05a69bd7d1d4bf291d7cda76658602e (DEFAULT_TABLE table=default.Table_510bc2b3752b42b09ebef315e1e79a48 [id=d3e4be6314074414a3c9ea93b53a14d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.817905 1487 tablet_service.cc:1552] Processing DeleteTablet for tablet 913fee0ca74e4cf99d6697ad5c38f4eb with delete_type TABLET_DATA_DELETED (replaced by tablet ca38f3a765264dcc83f73a5d586cfea9) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:24.822503 1337 tablet_service.cc:1505] Processing CreateTablet for tablet e58dc2f86c354a1dbc920e8f7872f027 (DEFAULT_TABLE table=default.Table_7cacdff7332349bca13cbc786bb5b346 [id=dae4c73664824c3e935bc663f866eef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.823748 1372 raft_consensus.cc:3060] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:24.823263 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 6491a33902bf46d093b611f82b70a5e6 (DEFAULT_TABLE table=default.Table_f3fbc16ea2444f798f1e23a68acdf61f [id=c6844395ab8c4a11801463f8a3248bdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.828725 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 913fee0ca74e4cf99d6697ad5c38f4eb with delete_type TABLET_DATA_DELETED (replaced by tablet ca38f3a765264dcc83f73a5d586cfea9) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:24.829787 1766 consensus_queue.cc:260] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.834666 1766 ts_tablet_manager.cc:1434] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.037s user 0.003s sys 0.000s
I20251024 19:48:24.835184 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541aebb900d6432dbf5aea81c7237120. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.835472 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6491a33902bf46d093b611f82b70a5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.835609 1766 tablet_bootstrap.cc:492] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:24.836236 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e58dc2f86c354a1dbc920e8f7872f027. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.836652 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05a69bd7d1d4bf291d7cda76658602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.840042 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da7d38b63708427982591da1014aca81" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:24.840530 1586 leader_election.cc:343] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.843148 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d1d3c977f9fe4fdfa9f638fa7ea69751"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:24.843760 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1d3c977f9fe4fdfa9f638fa7ea69751 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:24.847807 1767 tablet_bootstrap.cc:654] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.848407 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b445098bfd04729808641fcfb76927d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:24.850629 1308 leader_election.cc:343] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 65424b00ab9c454f9584048caba83e6a
I20251024 19:48:24.850776 1308 leader_election.cc:304] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:24.851044 3457 raft_consensus.cc:2749] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:24.852530 1774 ts_tablet_manager.cc:1929] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:24.861434 1611 tablet_service.cc:1505] Processing CreateTablet for tablet e14e63b6a2b44ff6a5a1b6e0d8873115 (DEFAULT_TABLE table=default.Table_de48e6da258b4f55b0167897017342fb [id=2a85232625454d4c8b1eb728c1e0102a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.862178 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e14e63b6a2b44ff6a5a1b6e0d8873115. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.862175 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet 7e747880c4e1425ba749201e624a083e from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:24.862841 1479 tablet_service.cc:1505] Processing CreateTablet for tablet e58dc2f86c354a1dbc920e8f7872f027 (DEFAULT_TABLE table=default.Table_7cacdff7332349bca13cbc786bb5b346 [id=dae4c73664824c3e935bc663f866eef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.863379 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e58dc2f86c354a1dbc920e8f7872f027. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.870112 1188 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet c215211de857468d9c28a9a87b29ef4f
I20251024 19:48:24.870510 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8f9e49a43cb94787a02a10d3017b4e75" } new_table_name: "default.Table_a72458e8d0914f539f32dd5323036661" modify_external_catalogs: true
I20251024 19:48:24.871953 1612 tablet_service.cc:1505] Processing CreateTablet for tablet e58dc2f86c354a1dbc920e8f7872f027 (DEFAULT_TABLE table=default.Table_7cacdff7332349bca13cbc786bb5b346 [id=dae4c73664824c3e935bc663f866eef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.872351 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e58dc2f86c354a1dbc920e8f7872f027. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.853435 1768 tablet_bootstrap.cc:492] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:24.863806 3396 leader_election.cc:290] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:24.879164 1768 ts_tablet_manager.cc:1403] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.195s user 0.001s sys 0.005s
I20251024 19:48:24.864118 1369 raft_consensus.cc:3060] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:24.879979 1768 raft_consensus.cc:359] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.880158 1768 raft_consensus.cc:385] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.880210 1768 raft_consensus.cc:740] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.880261 1776 ts_tablet_manager.cc:1929] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:24.880373 1776 log.cc:1199] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/59956dad06064024ab171631ef6c1dca
I20251024 19:48:24.880409 1768 consensus_queue.cc:260] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.880903 1768 ts_tablet_manager.cc:1434] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:24.881345 1768 tablet_bootstrap.cc:492] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:24.882342 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 6c49d87a219344fa866d41e95ec4934e (DEFAULT_TABLE table=default.Table_55de88bea3af4f0f8cc0c40bf6dbc127 [id=60f362b17cbf469e81d8d11ccf2901f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.882611 1774 log.cc:1199] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/1edc55caab904722bf19a1c71df601cf
I20251024 19:48:24.884445 1774 ts_tablet_manager.cc:1950] T 1edc55caab904722bf19a1c71df601cf P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:24.884841 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1048) took 1417 ms. Trace:
I20251024 19:48:24.876138 1487 tablet_service.cc:1505] Processing CreateTablet for tablet e14e63b6a2b44ff6a5a1b6e0d8873115 (DEFAULT_TABLE table=default.Table_de48e6da258b4f55b0167897017342fb [id=2a85232625454d4c8b1eb728c1e0102a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.884938 1774 rpcz_store.cc:276] 1024 19:48:23.467189 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.467269 (+ 80us) service_pool.cc:225] Handling call
1024 19:48:24.696426 (+1229157us) ts_tablet_manager.cc:1151] Deleting tablet 1edc55caab904722bf19a1c71df601cf
1024 19:48:24.696719 (+ 293us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.782060 (+ 85341us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.852423 (+ 70363us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.884820 (+ 32397us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1228922}
I20251024 19:48:24.885272 1774 tablet_replica.cc:333] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:24.885329 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e14e63b6a2b44ff6a5a1b6e0d8873115. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.885365 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4f22d5090caa447d9715e95a8a248432" } modify_external_catalogs: true
I20251024 19:48:24.885452 1774 raft_consensus.cc:2243] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:24.885552 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4f22d5090caa447d9715e95a8a248432" } modify_external_catalogs: true
I20251024 19:48:24.885649 1774 raft_consensus.cc:2272] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:24.886119 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c49d87a219344fa866d41e95ec4934e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.886248 1774 ts_tablet_manager.cc:1916] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:24.888247 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 1edc55caab904722bf19a1c71df601cf (table default.Table_641c979b063f419884a09e448c68a1b4 [id=b72bff5d78ec4eb08810442a084a16a8]) successfully deleted
I20251024 19:48:24.888731 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 6c49d87a219344fa866d41e95ec4934e (DEFAULT_TABLE table=default.Table_55de88bea3af4f0f8cc0c40bf6dbc127 [id=60f362b17cbf469e81d8d11ccf2901f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.889122 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c49d87a219344fa866d41e95ec4934e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.889190 1339 tablet_service.cc:1505] Processing CreateTablet for tablet e14e63b6a2b44ff6a5a1b6e0d8873115 (DEFAULT_TABLE table=default.Table_de48e6da258b4f55b0167897017342fb [id=2a85232625454d4c8b1eb728c1e0102a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.889478 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e14e63b6a2b44ff6a5a1b6e0d8873115. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.895192 1768 tablet_bootstrap.cc:654] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.896714 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 4bb850860d6641eba37cb7c7a8a2abc3 (DEFAULT_TABLE table=default.Table_fd2e2bba8e754ca3bdf8e7f36b01a40f [id=4cf9b4ba191447828716061b3b859456]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.897157 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb850860d6641eba37cb7c7a8a2abc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.898825 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 4bb850860d6641eba37cb7c7a8a2abc3 (DEFAULT_TABLE table=default.Table_fd2e2bba8e754ca3bdf8e7f36b01a40f [id=4cf9b4ba191447828716061b3b859456]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.906308 1767 tablet_bootstrap.cc:492] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:24.906716 1767 ts_tablet_manager.cc:1403] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.117s user 0.003s sys 0.002s
I20251024 19:48:24.907721 1767 raft_consensus.cc:359] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.908052 1767 raft_consensus.cc:385] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.908164 1767 raft_consensus.cc:740] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.908854 1776 ts_tablet_manager.cc:1950] T 59956dad06064024ab171631ef6c1dca P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:24.908946 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_61f1fbf85c954403a8374c6359cf52b9" } modify_external_catalogs: true
I20251024 19:48:24.909531 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_61f1fbf85c954403a8374c6359cf52b9" } modify_external_catalogs: true
I20251024 19:48:24.910778 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 067631b128fd4a1e9ab72c9507dab6c4 (table default.Table_814a6dad46d54945b70e652b7e0f0032 [id=3f394c59d82b44928beb22f42312a4a3]). Expected version 1 got 0
I20251024 19:48:24.911168 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet a1fbbba88a294adfbf97c0343d86698e (table default.Table_7bc8cd4a930b44dfafa473297875594d [id=a60c8222aa0248c091fab8dc34100847]). Expected version 2 got 0
I20251024 19:48:24.913070 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_11dceac7c7394c128fa1ede51ee9fdab" } new_table_name: "default.Table_3a34509a81134471b3977ce167d666ec" modify_external_catalogs: true
I20251024 19:48:24.913208 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 96a6e0a4882c42399fd07f06ac21aa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:24.914558 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 3e11f458b9194d029c9d109e9dad1d73 (DEFAULT_TABLE table=default.Table_3e8d42d7a60a43bf99d216729f8533ae [id=3c2c500b1db84554939155081938434b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.908986 1767 consensus_queue.cc:260] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.917243 1472 tablet_service.cc:1505] Processing CreateTablet for tablet 4bb850860d6641eba37cb7c7a8a2abc3 (DEFAULT_TABLE table=default.Table_fd2e2bba8e754ca3bdf8e7f36b01a40f [id=4cf9b4ba191447828716061b3b859456]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.917719 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb850860d6641eba37cb7c7a8a2abc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.918025 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:24.922811 1768 tablet_bootstrap.cc:492] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:24.919248 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 96a6e0a4882c42399fd07f06ac21aa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.922966 1768 ts_tablet_manager.cc:1403] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20251024 19:48:24.919833 1627 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b with delete_type TABLET_DATA_DELETED (replaced by tablet 6bebcdb80476423b8e6c750f749a1771) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.920522 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 45cff9bcc4924e0081518c284f074940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.920780 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 3e11f458b9194d029c9d109e9dad1d73 (DEFAULT_TABLE table=default.Table_3e8d42d7a60a43bf99d216729f8533ae [id=3c2c500b1db84554939155081938434b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.921468 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet da7d38b63708427982591da1014aca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.922152 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d26e02ad2945b39cf27738ed5b7cf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.922209 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 1edc55caab904722bf19a1c71df601cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.923710 1768 raft_consensus.cc:359] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.923882 1768 raft_consensus.cc:385] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.922330 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet d840888183ec4ef7bb245363c5bac322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.923931 1768 raft_consensus.cc:740] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.920099 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e7b3288182744969e217d4f174fe4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.919302 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet 38818c7997c84cedad5f9b09eef77105 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.924136 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e11f458b9194d029c9d109e9dad1d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.915143 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1058) took 3692 ms. Trace:
I20251024 19:48:24.924180 1768 consensus_queue.cc:260] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.924203 1776 rpcz_store.cc:276] 1024 19:48:21.223114 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.223172 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:24.790553 (+3567381us) ts_tablet_manager.cc:1151] Deleting tablet 59956dad06064024ab171631ef6c1dca
1024 19:48:24.790563 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.803792 (+ 13229us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.880145 (+ 76353us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.915120 (+ 34975us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3567202}
I20251024 19:48:24.924702 1768 ts_tablet_manager.cc:1434] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:24.925973 1346 tablet_service.cc:1505] Processing CreateTablet for tablet b6b1f0e1d2574a228458926581d473e3 (DEFAULT_TABLE table=default.Table_7aba73c5ba134980be24d93cae8e5f45 [id=623e322a745c481f93822713e6da0c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.915833 1767 ts_tablet_manager.cc:1434] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251024 19:48:24.927791 1372 raft_consensus.cc:3060] T f965111ac60841dbb9e6ecc21ff3fec0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:24.929741 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 59956dad06064024ab171631ef6c1dca (table default.Table_9c03ce6b892d44fcbff5c92f613bc37f [id=e51bd6230e2e41429af7e8b08198bbe1]) successfully deleted
I20251024 19:48:24.930150 1613 tablet_service.cc:1505] Processing CreateTablet for tablet b6b1f0e1d2574a228458926581d473e3 (DEFAULT_TABLE table=default.Table_7aba73c5ba134980be24d93cae8e5f45 [id=623e322a745c481f93822713e6da0c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.931164 1776 tablet_replica.cc:333] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:24.931422 1776 raft_consensus.cc:2243] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:24.931851 1776 raft_consensus.cc:2272] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:24.932523 1776 ts_tablet_manager.cc:1916] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:24.933909 1349 tablet_service.cc:1505] Processing CreateTablet for tablet fcee4c253eb84b9ab13cb664c4668a54 (DEFAULT_TABLE table=default.Table_a316e0ba22744f2c9b7431da3dbed677 [id=7672896f82594f7c88b8d6bb870a3310]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.934669 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6b1f0e1d2574a228458926581d473e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.936167 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 71ab984ce3124f58a059c5fd4dc29b55 (DEFAULT_TABLE table=default.Table_d1092b6bac8f4c90a5eb12dd16060570 [id=82df374b64b144d48259d165e3072544]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.936566 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ab984ce3124f58a059c5fd4dc29b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.937981 1342 tablet_service.cc:1505] Processing CreateTablet for tablet c1af38798ab14525ab77edae5aaf7092 (DEFAULT_TABLE table=default.Table_596ac12622974095b23e65278e8ba38b [id=898514395e4a4098a7bd097661c5e7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.938316 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1af38798ab14525ab77edae5aaf7092. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.938616 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 4100abc9ff12471e9a4959b6b24f7e99 (table default.Table_da24358ca9544a0e805bb937715888f9 [id=0e3c3f6ff360497c93fac944f351702c]). Expected version 1 got 0
I20251024 19:48:24.939689 1622 tablet_service.cc:1505] Processing CreateTablet for tablet c1af38798ab14525ab77edae5aaf7092 (DEFAULT_TABLE table=default.Table_596ac12622974095b23e65278e8ba38b [id=898514395e4a4098a7bd097661c5e7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.940377 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 23d4dd0b42d343cb9354ac698a8c761f with delete_type TABLET_DATA_DELETED (replaced by tablet fce7de9523fb4878a58aa8aef0c6b182) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.944454 1768 tablet_bootstrap.cc:492] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:24.945760 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 71ab984ce3124f58a059c5fd4dc29b55 (DEFAULT_TABLE table=default.Table_d1092b6bac8f4c90a5eb12dd16060570 [id=82df374b64b144d48259d165e3072544]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.946342 1768 tablet_bootstrap.cc:654] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.946652 3404 raft_consensus.cc:493] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.946727 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 1edc55caab904722bf19a1c71df601cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:24.946799 3404 raft_consensus.cc:515] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.947430 3404 leader_election.cc:290] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:24.947882 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cff9bcc4924e0081518c284f074940" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:24.949635 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 799780bf376843499c659df4cd2b4aa1 (DEFAULT_TABLE table=default.Table_434a6ef23bea4f3991418b6ed2d9f635 [id=f24ffafc3bb24a57a21ec7e070a46d03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.951792 1774 ts_tablet_manager.cc:1929] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:24.933544 1767 tablet_bootstrap.cc:492] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:24.934795 1611 tablet_service.cc:1505] Processing CreateTablet for tablet fcee4c253eb84b9ab13cb664c4668a54 (DEFAULT_TABLE table=default.Table_a316e0ba22744f2c9b7431da3dbed677 [id=7672896f82594f7c88b8d6bb870a3310]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.935081 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6b1f0e1d2574a228458926581d473e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.953312 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcee4c253eb84b9ab13cb664c4668a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.953745 1774 log.cc:1199] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/81f8393f78204860bd854f4872099387
W20251024 19:48:24.953769 1586 leader_election.cc:343] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.954517 1774 ts_tablet_manager.cc:1950] T 81f8393f78204860bd854f4872099387 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:24.955117 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1049) took 1487 ms. Trace:
I20251024 19:48:24.955355 1774 rpcz_store.cc:276] 1024 19:48:23.467918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.468034 (+ 116us) service_pool.cc:225] Handling call
1024 19:48:24.885215 (+1417181us) ts_tablet_manager.cc:1151] Deleting tablet 81f8393f78204860bd854f4872099387
1024 19:48:24.885227 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.948094 (+ 62867us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.951680 (+ 3586us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.955094 (+ 3414us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1417027}
I20251024 19:48:24.956154 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 81f8393f78204860bd854f4872099387 (table default.Table_2d74dcbcd9f249e5b10e1986cc0c0d18 [id=0bd294ee0c8341a6a29ce26caeeceb22]) successfully deleted
I20251024 19:48:24.956200 1774 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54174 (request call id 1052) took 1470 ms. Trace:
I20251024 19:48:24.956513 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1af38798ab14525ab77edae5aaf7092. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.956476 1774 rpcz_store.cc:276] 1024 19:48:23.486211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.486404 (+ 193us) service_pool.cc:225] Handling call
1024 19:48:24.955957 (+1469553us) ts_tablet_manager.cc:1151] Deleting tablet 2d8b8a1cff3540f7b7f13712425d534e
1024 19:48:24.955971 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.956161 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1469345}
I20251024 19:48:24.958217 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ab984ce3124f58a059c5fd4dc29b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.959214 1774 tablet_replica.cc:333] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:24.959553 1774 raft_consensus.cc:2243] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:24.952411 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e11f458b9194d029c9d109e9dad1d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.953516 1768 tablet_bootstrap.cc:492] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:24.960098 1768 ts_tablet_manager.cc:1403] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20251024 19:48:24.960893 1768 raft_consensus.cc:359] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.961078 1768 raft_consensus.cc:385] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.961138 1768 raft_consensus.cc:740] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.961395 1768 consensus_queue.cc:260] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.961910 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcee4c253eb84b9ab13cb664c4668a54. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:24.965792 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2d8b8a1cff3540f7b7f13712425d534e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d8b8a1cff3540f7b7f13712425d534e already in progress: creating tablet
W20251024 19:48:24.966061 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 03d8ab62b47f474198841ea14a3b7b00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03d8ab62b47f474198841ea14a3b7b00 already in progress: creating tablet
W20251024 19:48:24.966239 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 0a5034103c1e4e039df7fc9f0ab65d0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a5034103c1e4e039df7fc9f0ab65d0c already in progress: creating tablet
W20251024 19:48:24.966395 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 33d42f4c818340cda4ae615132fcf03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33d42f4c818340cda4ae615132fcf03f already in progress: creating tablet
W20251024 19:48:24.966611 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 9229fa6bf13441fa894e7bb7f78d7038 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9229fa6bf13441fa894e7bb7f78d7038 already in progress: creating tablet
W20251024 19:48:24.966764 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 91bb076b4b0d4b04955043248460d04c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91bb076b4b0d4b04955043248460d04c already in progress: creating tablet
W20251024 19:48:24.966907 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4b4904080b7a449fb92e773032808533 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b4904080b7a449fb92e773032808533 already in progress: creating tablet
W20251024 19:48:24.967083 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 913fee0ca74e4cf99d6697ad5c38f4eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 913fee0ca74e4cf99d6697ad5c38f4eb already in progress: creating tablet
I20251024 19:48:24.967733 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 24ff2839ff1e48d5a2f84890ec25c75b
I20251024 19:48:24.968904 1768 ts_tablet_manager.cc:1434] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251024 19:48:24.969352 1768 tablet_bootstrap.cc:492] T 1999b751e9344dcf81d26226c0589c8f P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:24.970703 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9dec6c29bd194b869699fef3b34923c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:24.971316 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dec6c29bd194b869699fef3b34923c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:24.973459 3404 raft_consensus.cc:493] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.973644 3404 raft_consensus.cc:515] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:24.974301 3404 leader_election.cc:290] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:24.974949 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ff2839ff1e48d5a2f84890ec25c75b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:24 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:24.974370 1768 tablet_bootstrap.cc:654] T 1999b751e9344dcf81d26226c0589c8f P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.959853 1774 raft_consensus.cc:2272] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:24.977357 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd8aa1d9aca429dbfe46f472ebbf53b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:24.977545 1774 ts_tablet_manager.cc:1916] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:24.977984 1586 leader_election.cc:343] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:24.980242 1767 tablet_bootstrap.cc:654] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:24.981519 1776 ts_tablet_manager.cc:1929] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:24.982298 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0dea9bad168942498e4ebfe3bf1eebd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:24.983227 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dea9bad168942498e4ebfe3bf1eebd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:24.977831 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799780bf376843499c659df4cd2b4aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.984405 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb850860d6641eba37cb7c7a8a2abc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.985224 1349 tablet_service.cc:1505] Processing CreateTablet for tablet d35f888748734ea39e342d1e5070e461 (DEFAULT_TABLE table=default.Table_b5f8df062a424c1f87d85918c22ebd13 [id=d5cda430b878446aa201b0f6b83dff90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:24.985621 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35f888748734ea39e342d1e5070e461. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.987090 3457 raft_consensus.cc:493] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.987231 3457 raft_consensus.cc:515] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.987882 3457 leader_election.cc:290] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:24.988377 1506 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eafde5a40df4fbfb03340b7bf791a2a" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:24.988677 1506 raft_consensus.cc:2393] T 4eafde5a40df4fbfb03340b7bf791a2a P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:24.992080 1774 ts_tablet_manager.cc:1929] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:24.982863 1768 tablet_bootstrap.cc:492] T 1999b751e9344dcf81d26226c0589c8f P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:24.978261 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ff2839ff1e48d5a2f84890ec25c75b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:24 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:24.992527 1768 ts_tablet_manager.cc:1403] T 1999b751e9344dcf81d26226c0589c8f P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.003s
W20251024 19:48:24.992990 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2851d58164d64370ba1f11be29fa1d33 consensus state has no leader
I20251024 19:48:24.993425 3390 raft_consensus.cc:493] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:24.993415 1768 raft_consensus.cc:359] T 1999b751e9344dcf81d26226c0589c8f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:24.993415 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_713a394d7b9f4e78b25f9569145f4d1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:24.993613 1768 raft_consensus.cc:385] T 1999b751e9344dcf81d26226c0589c8f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:24.993672 1768 raft_consensus.cc:740] T 1999b751e9344dcf81d26226c0589c8f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:24.993624 3390 raft_consensus.cc:515] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:24.993857 1768 consensus_queue.cc:260] T 1999b751e9344dcf81d26226c0589c8f P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:24.994076 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_713a394d7b9f4e78b25f9569145f4d1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:24.994333 1768 ts_tablet_manager.cc:1434] T 1999b751e9344dcf81d26226c0589c8f P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:24.994439 1774 log.cc:1199] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/96a6e0a4882c42399fd07f06ac21aa3f
I20251024 19:48:24.994813 1774 ts_tablet_manager.cc:1950] T 96a6e0a4882c42399fd07f06ac21aa3f P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:24.995020 3390 leader_election.cc:290] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.003006 1768 tablet_bootstrap.cc:492] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:25.004380 3455 raft_consensus.cc:493] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.004621 3455 raft_consensus.cc:515] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.004841 1768 tablet_bootstrap.cc:654] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:25.005021 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 018d3e6aba6740f0bd82e8c83863bc4e
I20251024 19:48:25.006209 3455 leader_election.cc:290] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.006178 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 96a6e0a4882c42399fd07f06ac21aa3f (table default.Table_19bbc037a7f241f5b338f4cfa4ebe82d [id=b2ce46e067994c9fa4df9f00de59d68f]) successfully deleted
W20251024 19:48:25.006412 1179 catalog_manager.cc:4952] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 1edc55caab904722bf19a1c71df601cf because the tablet was not found. No further retry: Not found: Tablet not found: 1edc55caab904722bf19a1c71df601cf
W20251024 19:48:25.006613 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 24ff2839ff1e48d5a2f84890ec25c75b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ff2839ff1e48d5a2f84890ec25c75b already in progress: creating tablet
I20251024 19:48:25.008395 1766 tablet_bootstrap.cc:654] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:25.008841 1503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cff9bcc4924e0081518c284f074940" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:25.008999 1503 raft_consensus.cc:2468] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:25.009366 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8631b5d780614ecdb746a74a4d97b8b5" } new_table_name: "default.Table_d7da4325d747469b86ff66b85690867d" modify_external_catalogs: true
I20251024 19:48:25.009670 3455 raft_consensus.cc:493] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.009727 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd8aa1d9aca429dbfe46f472ebbf53b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:25.009888 1505 raft_consensus.cc:2468] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:25.009860 3455 raft_consensus.cc:515] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.010636 1486 tablet_service.cc:1505] Processing CreateTablet for tablet b6b1f0e1d2574a228458926581d473e3 (DEFAULT_TABLE table=default.Table_7aba73c5ba134980be24d93cae8e5f45 [id=623e322a745c481f93822713e6da0c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.010732 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e38ef2ac19d14f9c91fa8abe86cc7987" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.011052 1486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6b1f0e1d2574a228458926581d473e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.011121 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cff9bcc4924e0081518c284f074940" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.011895 3455 leader_election.cc:290] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.013391 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 018d3e6aba6740f0bd82e8c83863bc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:24 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.021970 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 6c49d87a219344fa866d41e95ec4934e (DEFAULT_TABLE table=default.Table_55de88bea3af4f0f8cc0c40bf6dbc127 [id=60f362b17cbf469e81d8d11ccf2901f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.024343 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1024) took 4817 ms. Trace:
I20251024 19:48:25.024426 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 71ab984ce3124f58a059c5fd4dc29b55 (DEFAULT_TABLE table=default.Table_d1092b6bac8f4c90a5eb12dd16060570 [id=82df374b64b144d48259d165e3072544]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.024881 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ab984ce3124f58a059c5fd4dc29b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.029430 1488 tablet_service.cc:1505] Processing CreateTablet for tablet fcee4c253eb84b9ab13cb664c4668a54 (DEFAULT_TABLE table=default.Table_a316e0ba22744f2c9b7431da3dbed677 [id=7672896f82594f7c88b8d6bb870a3310]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.029604 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 7e747880c4e1425ba749201e624a083e
I20251024 19:48:25.029671 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 3e11f458b9194d029c9d109e9dad1d73 (DEFAULT_TABLE table=default.Table_3e8d42d7a60a43bf99d216729f8533ae [id=3c2c500b1db84554939155081938434b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.029853 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcee4c253eb84b9ab13cb664c4668a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.030095 1184 catalog_manager.cc:6719] ReplaceTablet: tablet 7e747880c4e1425ba749201e624a083e deleted and replaced by tablet 972f9ca0d2234a5f8d698090a6825bfb
I20251024 19:48:25.030180 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e11f458b9194d029c9d109e9dad1d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.031057 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c49d87a219344fa866d41e95ec4934e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.035822 1473 tablet_service.cc:1505] Processing CreateTablet for tablet c1af38798ab14525ab77edae5aaf7092 (DEFAULT_TABLE table=default.Table_596ac12622974095b23e65278e8ba38b [id=898514395e4a4098a7bd097661c5e7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.036235 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ff2839ff1e48d5a2f84890ec25c75b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:24 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:25.040149 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 018d3e6aba6740f0bd82e8c83863bc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 018d3e6aba6740f0bd82e8c83863bc4e already in progress: creating tablet
I20251024 19:48:25.042531 1768 tablet_bootstrap.cc:492] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:25.042800 1768 ts_tablet_manager.cc:1403] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.004s
I20251024 19:48:25.042832 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d1c0f808bdec4b94825c72f0043421cc" } new_table_name: "default.Table_c21567dbbfa84909bb658571e5cc7214" modify_external_catalogs: true
I20251024 19:48:25.043660 1768 raft_consensus.cc:359] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.043844 1768 raft_consensus.cc:385] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:25.043900 1768 raft_consensus.cc:740] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
W20251024 19:48:25.044087 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0276f82a2e78458f9d065c94c6d32edc consensus state has no leader
I20251024 19:48:25.044104 1768 consensus_queue.cc:260] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.044869 1768 ts_tablet_manager.cc:1434] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251024 19:48:25.045147 1768 tablet_bootstrap.cc:492] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:25.047147 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 8f46974e56c64a58a9c562b235ff8cfe
I20251024 19:48:25.047930 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f46974e56c64a58a9c562b235ff8cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:24 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.048696 3455 raft_consensus.cc:493] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251024 19:48:25.049086 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 8f46974e56c64a58a9c562b235ff8cfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f46974e56c64a58a9c562b235ff8cfe already in progress: creating tablet
I20251024 19:48:25.049564 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 6c887b0df24c45d894ca10d2eda71444
W20251024 19:48:25.050786 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc398cfc57084d6295a56dd3527fbfa5 consensus state has no leader
I20251024 19:48:25.050940 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c887b0df24c45d894ca10d2eda71444 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:24 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.051798 3455 raft_consensus.cc:515] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.053771 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (table default.Table_18e7285b7df44399ab3be1dfc86eb91f [id=14bda7f9324f44aa9cb87a5f129039be]). Expected version 2 got 0
I20251024 19:48:25.055668 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_eee1b7268a6147b1b8cc5ec4a8af199a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:25.055842 1349 tablet_service.cc:1552] Processing DeleteTablet for tablet 65424b00ab9c454f9584048caba83e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.055867 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet 250f106664be4b5aaa330dbb4e43cb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:25.056318 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eee1b7268a6147b1b8cc5ec4a8af199a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.056572 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_da73cd6e87304788ad9da534dbc041a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.057049 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da73cd6e87304788ad9da534dbc041a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.058077 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_430f5855fcbc49a39009fcc1f6bc115f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.058234 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 6c887b0df24c45d894ca10d2eda71444 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c887b0df24c45d894ca10d2eda71444 already in progress: creating tablet
I20251024 19:48:25.058305 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_05dbfb26c73a4c5aaff27e72dccdf03f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.058555 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_430f5855fcbc49a39009fcc1f6bc115f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.058609 1774 tablet_replica.cc:333] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
W20251024 19:48:25.058655 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05dbfb26c73a4c5aaff27e72dccdf03f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.058876 1774 raft_consensus.cc:2243] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:25.059048 1774 raft_consensus.cc:2272] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:25.059741 1768 tablet_bootstrap.cc:654] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:25.062806 1774 ts_tablet_manager.cc:1916] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:25.064321 1768 tablet_bootstrap.cc:492] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:25.064610 1768 ts_tablet_manager.cc:1403] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.001s
I20251024 19:48:25.066377 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 1999b751e9344dcf81d26226c0589c8f (table default.Table_9147a98b3bf9483ea7b7a6f17130d4fc [id=d743705c21da4b89abedf2eb34ddbe09]). Expected version 1 got 0
I20251024 19:48:25.066354 1768 raft_consensus.cc:359] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.066816 1768 raft_consensus.cc:385] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:25.066975 1768 raft_consensus.cc:740] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:25.068444 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet 38818c7997c84cedad5f9b09eef77105 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.067402 1768 consensus_queue.cc:260] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.069741 1768 ts_tablet_manager.cc:1434] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:48:25.070358 1768 tablet_bootstrap.cc:492] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:25.070575 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0228067f17b24fc0b45df5fd3f970997"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.071204 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0228067f17b24fc0b45df5fd3f970997 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.077950 3455 leader_election.cc:290] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.078198 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 018d3e6aba6740f0bd82e8c83863bc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:24 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.078622 1475 tablet_service.cc:1505] Processing CreateTablet for tablet d35f888748734ea39e342d1e5070e461 (DEFAULT_TABLE table=default.Table_b5f8df062a424c1f87d85918c22ebd13 [id=d5cda430b878446aa201b0f6b83dff90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.079267 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e747880c4e1425ba749201e624a083e with delete_type TABLET_DATA_DELETED (replaced by tablet 972f9ca0d2234a5f8d698090a6825bfb) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.079552 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1af38798ab14525ab77edae5aaf7092. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:24.983569 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 799780bf376843499c659df4cd2b4aa1 (DEFAULT_TABLE table=default.Table_434a6ef23bea4f3991418b6ed2d9f635 [id=f24ffafc3bb24a57a21ec7e070a46d03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.079727 1776 log.cc:1199] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/f3d80885fcc642b8a75da30cd4abecbd
I20251024 19:48:25.079970 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799780bf376843499c659df4cd2b4aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.080137 1776 ts_tablet_manager.cc:1950] T f3d80885fcc642b8a75da30cd4abecbd P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:25.080611 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35f888748734ea39e342d1e5070e461. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.082010 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1057) took 3859 ms. Trace:
I20251024 19:48:25.082175 1776 rpcz_store.cc:276] 1024 19:48:21.222970 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.223429 (+ 459us) service_pool.cc:225] Handling call
1024 19:48:24.925799 (+3702370us) ts_tablet_manager.cc:1151] Deleting tablet f3d80885fcc642b8a75da30cd4abecbd
1024 19:48:24.931089 (+ 5290us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:24.973039 (+ 41950us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:24.981396 (+ 8357us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:25.081957 (+100561us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3702211}
I20251024 19:48:25.082715 1776 tablet_replica.cc:333] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:25.082935 1776 raft_consensus.cc:2243] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:25.083106 1776 raft_consensus.cc:2272] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:25.083662 1776 ts_tablet_manager.cc:1916] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:24.988354 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4eafde5a40df4fbfb03340b7bf791a2a" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:24.994673 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e38ef2ac19d14f9c91fa8abe86cc7987" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:25.001029 1611 tablet_service.cc:1505] Processing CreateTablet for tablet d35f888748734ea39e342d1e5070e461 (DEFAULT_TABLE table=default.Table_b5f8df062a424c1f87d85918c22ebd13 [id=d5cda430b878446aa201b0f6b83dff90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.005925 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cff9bcc4924e0081518c284f074940" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:25.086263 1642 raft_consensus.cc:2468] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:25.086382 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet f3d80885fcc642b8a75da30cd4abecbd (table default.Table_7966ff9879d24254aed39562b2f127d7 [id=45b38d5e9fd6472fa318b41863080708]) successfully deleted
I20251024 19:48:25.086498 1305 leader_election.cc:304] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.086722 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35f888748734ea39e342d1e5070e461. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.011247 1586 leader_election.cc:304] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
W20251024 19:48:25.087121 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 75907.7us
I20251024 19:48:25.087954 1364 raft_consensus.cc:3060] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.087954 1586 leader_election.cc:304] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:25.088173 3360 raft_consensus.cc:2804] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:25.088239 3360 raft_consensus.cc:493] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:25.088299 3360 raft_consensus.cc:3060] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:25.088364 1586 consensus_peers.cc:597] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65424b00ab9c454f9584048caba83e6a. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:25.088626 3141 consensus_queue.cc:1048] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:25.089008 3141 raft_consensus.cc:2804] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:25.089067 3141 raft_consensus.cc:493] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:25.089118 3141 raft_consensus.cc:3060] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.091090 3360 raft_consensus.cc:515] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.091794 3360 leader_election.cc:290] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.013005 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23d4dd0b42d343cb9354ac698a8c761f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:25.092137 1649 raft_consensus.cc:2468] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:25.016745 1619 tablet_service.cc:1552] Processing DeleteTablet for tablet 018d3e6aba6740f0bd82e8c83863bc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:24 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.092306 3396 raft_consensus.cc:2749] T 4eafde5a40df4fbfb03340b7bf791a2a P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:25.012172 3387 consensus_queue.cc:1048] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:25.050758 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c887b0df24c45d894ca10d2eda71444 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:24 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:25.093163 1444 leader_election.cc:343] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.093360 1444 leader_election.cc:304] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:25.093312 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_84aeb3904e8f490c85eb6a322e05d144" } modify_external_catalogs: true
I20251024 19:48:25.093528 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_84aeb3904e8f490c85eb6a322e05d144" } modify_external_catalogs: true
I20251024 19:48:25.094228 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e7b3288182744969e217d4f174fe4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.094465 3455 raft_consensus.cc:2804] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:25.094429 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 799780bf376843499c659df4cd2b4aa1 (DEFAULT_TABLE table=default.Table_434a6ef23bea4f3991418b6ed2d9f635 [id=f24ffafc3bb24a57a21ec7e070a46d03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.094549 3455 raft_consensus.cc:493] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:25.094619 3455 raft_consensus.cc:3060] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.094825 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799780bf376843499c659df4cd2b4aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.031692 1622 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e747880c4e1425ba749201e624a083e with delete_type TABLET_DATA_DELETED (replaced by tablet 972f9ca0d2234a5f8d698090a6825bfb) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.079120 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58a8f4f90f794e83a158e0026e5cea7f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:25.097165 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e747880c4e1425ba749201e624a083e with delete_type TABLET_DATA_DELETED (replaced by tablet 972f9ca0d2234a5f8d698090a6825bfb) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.098227 1767 tablet_bootstrap.cc:492] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:25.098385 1767 ts_tablet_manager.cc:1403] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.165s user 0.004s sys 0.000s
I20251024 19:48:25.091759 3141 raft_consensus.cc:515] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:25.099261 3404 consensus_queue.cc:1048] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:25.099670 3404 consensus_queue.cc:1048] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:25.099988 3404 consensus_queue.cc:1048] T 4eafde5a40df4fbfb03340b7bf791a2a P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:25.100224 3141 leader_election.cc:290] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:25.097452 3455 raft_consensus.cc:515] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.100922 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f46974e56c64a58a9c562b235ff8cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:24 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.101141 1767 raft_consensus.cc:359] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.101327 1767 raft_consensus.cc:385] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251024 19:48:25.101336 1447 leader_election.cc:343] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.101384 1767 raft_consensus.cc:740] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:25.101460 1447 leader_election.cc:304] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:25.101902 1447 leader_election.cc:343] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.102123 1477 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c887b0df24c45d894ca10d2eda71444 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:24 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.102638 3455 leader_election.cc:290] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.103013 3455 raft_consensus.cc:2749] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:25.103119 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd8aa1d9aca429dbfe46f472ebbf53b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:25.103300 1505 raft_consensus.cc:3060] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.104792 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58a8f4f90f794e83a158e0026e5cea7f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.105600 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cff9bcc4924e0081518c284f074940" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:25.105980 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd8aa1d9aca429dbfe46f472ebbf53b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:25.106683 1586 consensus_peers.cc:597] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:25.106819 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23d4dd0b42d343cb9354ac698a8c761f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:25.106912 1586 leader_election.cc:343] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:25.107076 1586 leader_election.cc:343] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.105345 3360 consensus_queue.cc:1048] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251024 19:48:25.051926 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f46974e56c64a58a9c562b235ff8cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:24 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.115834 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b7f521e0feb04a5fa8c1475090117699"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.116463 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7f521e0feb04a5fa8c1475090117699 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:25.117209 1447 leader_election.cc:343] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.118021 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cff9bcc4924e0081518c284f074940" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:25.120348 1369 raft_consensus.cc:3060] T 1999b751e9344dcf81d26226c0589c8f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.120947 3390 raft_consensus.cc:493] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.121217 3390 raft_consensus.cc:515] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.122202 3390 leader_election.cc:290] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.122681 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 0266cef4e2034303978d7456c47ea98c with delete_type TABLET_DATA_DELETED (replaced by tablet 64325d89db714554866e74922d103e34) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.116488 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cff9bcc4924e0081518c284f074940" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:25.123746 1507 raft_consensus.cc:2393] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
I20251024 19:48:25.024466 1775 rpcz_store.cc:276] 1024 19:48:20.207228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.207280 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:25.007789 (+4800509us) ts_tablet_manager.cc:1151] Deleting tablet 6b1a3523954448de93357510e6c1ec4e
1024 19:48:25.007801 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.024320 (+ 16519us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4800343}
I20251024 19:48:25.124663 1775 tablet_replica.cc:333] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:25.124948 1775 raft_consensus.cc:2243] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:25.125281 1775 raft_consensus.cc:2272] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:25.126966 1775 ts_tablet_manager.cc:1916] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:25.093704 1444 leader_election.cc:304] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:25.129792 2858 raft_consensus.cc:2804] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:25.129881 2858 raft_consensus.cc:493] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:25.129946 2858 raft_consensus.cc:3060] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:25.129983 1444 leader_election.cc:343] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.130100 1444 leader_election.cc:304] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.130339 3455 raft_consensus.cc:2749] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:25.130364 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cff9bcc4924e0081518c284f074940" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:25.130620 1646 raft_consensus.cc:2393] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10a9536207e343f188e0ef709081d04e in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:25.130892 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "786fe2fd6fca407e83a76f74f96a8b16" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:25.131464 1444 leader_election.cc:343] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.131803 1586 leader_election.cc:304] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
W20251024 19:48:25.131845 1447 leader_election.cc:343] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.132548 3360 raft_consensus.cc:2749] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251024 19:48:25.137185 1366 raft_consensus.cc:3060] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.140609 3387 consensus_queue.cc:1048] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:25.156642 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 6b1a3523954448de93357510e6c1ec4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b1a3523954448de93357510e6c1ec4e already in progress: creating tablet
W20251024 19:48:25.157353 1447 leader_election.cc:343] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.158324 1447 leader_election.cc:304] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.158716 3455 raft_consensus.cc:2749] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20251024 19:48:25.159560 3462 log.cc:927] Time spent T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
W20251024 19:48:25.164749 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1fc0f99ca5c42bfb9663e21a7b761c9 consensus state has no leader
I20251024 19:48:25.165846 3141 raft_consensus.cc:493] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.166010 3141 raft_consensus.cc:515] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.166671 3141 leader_election.cc:290] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:25.173198 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2960211859cd450cba83877622d88cad consensus state has no leader
I20251024 19:48:25.174125 1767 consensus_queue.cc:260] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.175293 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_da24358ca9544a0e805bb937715888f9" } modify_external_catalogs: true
I20251024 19:48:25.175494 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_da24358ca9544a0e805bb937715888f9" } modify_external_catalogs: true
I20251024 19:48:25.175932 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "786fe2fd6fca407e83a76f74f96a8b16" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:25.178104 1447 leader_election.cc:343] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.178288 1447 leader_election.cc:304] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.178576 3456 raft_consensus.cc:2749] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:25.176888 1586 leader_election.cc:343] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 96a6e0a4882c42399fd07f06ac21aa3f
I20251024 19:48:25.190652 1370 raft_consensus.cc:3060] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.190959 1774 ts_tablet_manager.cc:1929] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:25.191150 1774 log.cc:1199] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/65424b00ab9c454f9584048caba83e6a
I20251024 19:48:25.192469 1774 ts_tablet_manager.cc:1950] T 65424b00ab9c454f9584048caba83e6a P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:25.193377 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 65424b00ab9c454f9584048caba83e6a (table default.Table_e6ec362938cc478d9b9abee36f34f5c5 [id=d2e2211a2cd8457aab95305acead0b3b]) successfully deleted
I20251024 19:48:25.194015 1774 tablet_replica.cc:333] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
W20251024 19:48:25.194520 1586 leader_election.cc:343] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 96a6e0a4882c42399fd07f06ac21aa3f
I20251024 19:48:25.194648 1586 leader_election.cc:304] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:25.195041 1774 raft_consensus.cc:2243] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:25.195577 3360 raft_consensus.cc:2749] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:25.196110 1774 raft_consensus.cc:2272] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:25.197746 1774 ts_tablet_manager.cc:1916] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:25.198177 3360 consensus_queue.cc:1048] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:25.201628 1586 consensus_peers.cc:597] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:25.203770 1767 ts_tablet_manager.cc:1434] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.105s user 0.001s sys 0.001s
I20251024 19:48:25.207002 3404 consensus_queue.cc:1048] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:25.214247 3360 consensus_queue.cc:1048] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:25.215521 1767 tablet_bootstrap.cc:492] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:25.216925 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_33762464873048acb73f8d26e167301e" } new_table_name: "default.Table_d42d47ff1c304c3bbfa2173c50f53728" modify_external_catalogs: true
I20251024 19:48:25.225310 1768 tablet_bootstrap.cc:654] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:25.226223 3387 raft_consensus.cc:493] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.226403 3387 raft_consensus.cc:515] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.227056 3387 leader_election.cc:290] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.230293 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4100abc9ff12471e9a4959b6b24f7e99" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:25.230841 1586 leader_election.cc:343] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.232373 3456 raft_consensus.cc:493] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.232632 3456 raft_consensus.cc:515] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.236570 1767 tablet_bootstrap.cc:654] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:25.243105 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_40bc5f8e5a2d4aeeb9ce677ed9dd0ccb" } modify_external_catalogs: true
I20251024 19:48:25.243295 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_40bc5f8e5a2d4aeeb9ce677ed9dd0ccb" } modify_external_catalogs: true
I20251024 19:48:25.249810 1505 raft_consensus.cc:2468] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:25.254846 1767 tablet_bootstrap.cc:492] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:25.255013 1767 ts_tablet_manager.cc:1403] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.003s
I20251024 19:48:25.256024 1767 raft_consensus.cc:359] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:25.256248 1767 raft_consensus.cc:385] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:25.256299 1767 raft_consensus.cc:740] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:25.259627 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b48023645eb94ba4a0341afe616222b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.260241 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b48023645eb94ba4a0341afe616222b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.264245 1586 leader_election.cc:304] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:25.264523 3141 consensus_queue.cc:1048] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:25.264928 3141 raft_consensus.cc:2804] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:25.265046 3141 raft_consensus.cc:697] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:25.264019 1506 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4100abc9ff12471e9a4959b6b24f7e99" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:25.267428 1586 consensus_peers.cc:597] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:25.268316 3455 raft_consensus.cc:493] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.270040 3455 raft_consensus.cc:515] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.270735 3455 leader_election.cc:290] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.270713 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5be196925fd1419998ee4e85f267f8a4" } new_table_name: "default.Table_069b7152d03c452e930acc5ed4cec256" modify_external_catalogs: true
I20251024 19:48:25.271943 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1cdd5ec32ca49c1bfafaf3a24050e89" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:25.273241 1444 leader_election.cc:343] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.273331 1767 consensus_queue.cc:260] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:25.273949 1767 ts_tablet_manager.cc:1434] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.019s user 0.000s sys 0.003s
I20251024 19:48:25.274744 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9ba52b13507546c4b00c9109363aacc9" } modify_external_catalogs: true
I20251024 19:48:25.274916 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9ba52b13507546c4b00c9109363aacc9" } modify_external_catalogs: true
I20251024 19:48:25.275225 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9b49c126d44b4afeb3d1122999d2b8da" } modify_external_catalogs: true
I20251024 19:48:25.275358 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9b49c126d44b4afeb3d1122999d2b8da" } modify_external_catalogs: true
I20251024 19:48:25.276535 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1cdd5ec32ca49c1bfafaf3a24050e89" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:25.277199 1447 leader_election.cc:343] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.277331 1447 leader_election.cc:304] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.277958 3455 raft_consensus.cc:2749] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:25.278975 2858 raft_consensus.cc:515] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.282109 1506 raft_consensus.cc:2468] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:25.283484 3456 leader_election.cc:290] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.283699 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee3c521876324808857f7724028c895f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.284291 2858 leader_election.cc:290] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.275233 3141 consensus_queue.cc:237] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:25.285467 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23d4dd0b42d343cb9354ac698a8c761f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:25.285697 1646 raft_consensus.cc:3060] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.288727 1646 raft_consensus.cc:2468] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:25.283083 1586 leader_election.cc:304] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:25.282711 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee3c521876324808857f7724028c895f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:25.289387 3387 raft_consensus.cc:2804] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:25.289484 3387 raft_consensus.cc:493] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:25.289567 3387 raft_consensus.cc:3060] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.290052 1444 leader_election.cc:304] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:25.290344 1444 leader_election.cc:343] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.290951 2858 raft_consensus.cc:2804] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:25.291064 2858 raft_consensus.cc:697] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:25.291277 2858 consensus_queue.cc:237] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.295166 3387 raft_consensus.cc:515] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.284828 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23d4dd0b42d343cb9354ac698a8c761f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:25.302748 1447 leader_election.cc:343] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:25.303390 1447 leader_election.cc:343] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.303515 1447 leader_election.cc:304] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.303845 3390 raft_consensus.cc:2749] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:25.306430 3457 raft_consensus.cc:493] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.306602 3457 raft_consensus.cc:515] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.307278 3457 leader_election.cc:290] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:25.308629 1506 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0266cef4e2034303978d7456c47ea98c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:25.308768 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet b632acc1348e49f8a1eea019c007780c (table default.Table_361840e201d943c18014ec71a255cc9c [id=dc00ee30038e49fb9f86499cfb985a0e]). Expected version 2 got 0
I20251024 19:48:25.311574 1186 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 58a8f4f90f794e83a158e0026e5cea7f (table default.Table_c7b7dc69f2e640f6bb21208e8d8f32b8 [id=3fe81a9c4a134e908f4f2599c4b0cbf4]). Expected version 1 got 0
I20251024 19:48:25.311757 1186 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet a1fbbba88a294adfbf97c0343d86698e (table default.Table_7bc8cd4a930b44dfafa473297875594d [id=a60c8222aa0248c091fab8dc34100847]). Expected version 2 got 0
I20251024 19:48:25.311929 1186 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 2b2992a738b440d798f5e61cc45ec088 (table default.Table_8523c29419f0445aa4da2ffdc6480f1a [id=40394401a8a84431bfb5e2b1db774832]). Expected version 4 got 0
I20251024 19:48:25.314337 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_289bdf22b5834ac78a2b65cd33fac990" } new_table_name: "default.Table_7c96d7be6da64c3f92be858415e51c1f" modify_external_catalogs: true
I20251024 19:48:25.314682 1351 tablet_service.cc:1505] Processing CreateTablet for tablet a3995aca07e9434c9ac002d57b75cd5c (DEFAULT_TABLE table=default.Table_c0b98ba82d324740b9152d554bea55d3 [id=e2236e946c8a4eb08aa0b6d0a3222593]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.315078 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3995aca07e9434c9ac002d57b75cd5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.315675 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 578f14ab0f5d4914a8ab49754d309f14 (DEFAULT_TABLE table=default.Table_2c4240fa3c4d4fa5bd42eeb197a2088d [id=06a667007123416cb62e89aa6f6feedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.315974 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578f14ab0f5d4914a8ab49754d309f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.317834 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 089c07fc56e64ef4a21e0c0f6879e393 (DEFAULT_TABLE table=default.Table_f8fd3d3a919d4e91844c5ded89bced1b [id=00d5653db09b4574899090f1522b6ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.318163 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089c07fc56e64ef4a21e0c0f6879e393. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.319130 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6b368f6653044b0bb1f7062a5ffa4209"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.319648 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b368f6653044b0bb1f7062a5ffa4209 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.320556 2858 raft_consensus.cc:493] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.320720 2858 raft_consensus.cc:515] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.323374 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 7d026685931f4cf380375a1fce1312c7 (DEFAULT_TABLE table=default.Table_2b21687caa4b4a0595f99684d3d95b22 [id=664f63f566024591aa3619caab5dcb63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.323752 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d026685931f4cf380375a1fce1312c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.325438 2858 leader_election.cc:290] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.327122 1338 tablet_service.cc:1505] Processing CreateTablet for tablet ec6d23415f28455d804dd770cb1218b3 (DEFAULT_TABLE table=default.Table_3db01f386bf3464cbfee13a63abf924a [id=777eec9a388a4513bd96f34e5c6a5661]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.327776 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec6d23415f28455d804dd770cb1218b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.334021 3387 leader_election.cc:290] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.335089 1347 tablet_service.cc:1505] Processing CreateTablet for tablet fc4e9e3d4725427ea3fc66ec4f183ffb (DEFAULT_TABLE table=default.Table_069227ac82b5485b9f88980affdf85ba [id=44460f6ebe164940a2b5adc7f2a5e9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.335577 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc4e9e3d4725427ea3fc66ec4f183ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.338538 1774 ts_tablet_manager.cc:1929] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:25.347079 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 7145915fe8f94d52868d9b33a7149efa (DEFAULT_TABLE table=default.Table_5be196925fd1419998ee4e85f267f8a4 [id=1a1bcc9018ff4c31b69cd8791fe7e4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.347590 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7145915fe8f94d52868d9b33a7149efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.348762 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4100abc9ff12471e9a4959b6b24f7e99" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:25.348942 1510 raft_consensus.cc:3060] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.349866 2858 raft_consensus.cc:493] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.349987 2858 raft_consensus.cc:515] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.350600 2858 leader_election.cc:290] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.352759 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a69c0918d2e4eed9d91b1913ec2c21e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:25.354729 1444 leader_election.cc:343] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:25.356258 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fbbca3bad5f4b448ae5afc4d5cfed14 consensus state has no leader
W20251024 19:48:25.357827 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7145915fe8f94d52868d9b33a7149efa consensus state has no leader
I20251024 19:48:25.364166 1485 tablet_service.cc:1505] Processing CreateTablet for tablet fc4e9e3d4725427ea3fc66ec4f183ffb (DEFAULT_TABLE table=default.Table_069227ac82b5485b9f88980affdf85ba [id=44460f6ebe164940a2b5adc7f2a5e9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.366200 1487 tablet_service.cc:1505] Processing CreateTablet for tablet a3995aca07e9434c9ac002d57b75cd5c (DEFAULT_TABLE table=default.Table_c0b98ba82d324740b9152d554bea55d3 [id=e2236e946c8a4eb08aa0b6d0a3222593]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.366546 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3995aca07e9434c9ac002d57b75cd5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.366815 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 578f14ab0f5d4914a8ab49754d309f14 (DEFAULT_TABLE table=default.Table_2c4240fa3c4d4fa5bd42eeb197a2088d [id=06a667007123416cb62e89aa6f6feedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.367110 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578f14ab0f5d4914a8ab49754d309f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.368813 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 64499c8734d34c66b0b52a2cd3ef0673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.369318 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 786fe2fd6fca407e83a76f74f96a8b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.369733 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc4e9e3d4725427ea3fc66ec4f183ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.365804 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 7d026685931f4cf380375a1fce1312c7 (DEFAULT_TABLE table=default.Table_2b21687caa4b4a0595f99684d3d95b22 [id=664f63f566024591aa3619caab5dcb63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.274279 1767 tablet_bootstrap.cc:492] T ccd46da86fe444baa675d3391cdd410c P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:25.375177 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a69c0918d2e4eed9d91b1913ec2c21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.375631 1482 tablet_service.cc:1552] Processing DeleteTablet for tablet e38ef2ac19d14f9c91fa8abe86cc7987 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1a3523954448de93357510e6c1ec4e) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.377369 1488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d026685931f4cf380375a1fce1312c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.305071 1776 ts_tablet_manager.cc:1929] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:25.311257 1611 tablet_service.cc:1505] Processing CreateTablet for tablet fc4e9e3d4725427ea3fc66ec4f183ffb (DEFAULT_TABLE table=default.Table_069227ac82b5485b9f88980affdf85ba [id=44460f6ebe164940a2b5adc7f2a5e9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.379987 1482 tablet_service.cc:1505] Processing CreateTablet for tablet 089c07fc56e64ef4a21e0c0f6879e393 (DEFAULT_TABLE table=default.Table_f8fd3d3a919d4e91844c5ded89bced1b [id=00d5653db09b4574899090f1522b6ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.314793 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 578f14ab0f5d4914a8ab49754d309f14 (DEFAULT_TABLE table=default.Table_2c4240fa3c4d4fa5bd42eeb197a2088d [id=06a667007123416cb62e89aa6f6feedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.380363 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089c07fc56e64ef4a21e0c0f6879e393. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.380497 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578f14ab0f5d4914a8ab49754d309f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.381363 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 7145915fe8f94d52868d9b33a7149efa (DEFAULT_TABLE table=default.Table_5be196925fd1419998ee4e85f267f8a4 [id=1a1bcc9018ff4c31b69cd8791fe7e4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.382776 1489 tablet_service.cc:1505] Processing CreateTablet for tablet ec6d23415f28455d804dd770cb1218b3 (DEFAULT_TABLE table=default.Table_3db01f386bf3464cbfee13a63abf924a [id=777eec9a388a4513bd96f34e5c6a5661]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.383076 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec6d23415f28455d804dd770cb1218b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.315101 1613 tablet_service.cc:1505] Processing CreateTablet for tablet a3995aca07e9434c9ac002d57b75cd5c (DEFAULT_TABLE table=default.Table_c0b98ba82d324740b9152d554bea55d3 [id=e2236e946c8a4eb08aa0b6d0a3222593]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.386588 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3995aca07e9434c9ac002d57b75cd5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.315315 1649 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0266cef4e2034303978d7456c47ea98c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:25.315867 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 7d026685931f4cf380375a1fce1312c7 (DEFAULT_TABLE table=default.Table_2b21687caa4b4a0595f99684d3d95b22 [id=664f63f566024591aa3619caab5dcb63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.390524 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d026685931f4cf380375a1fce1312c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.391966 1774 log.cc:1199] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/250f106664be4b5aaa330dbb4e43cb7b
I20251024 19:48:25.392339 1774 ts_tablet_manager.cc:1950] T 250f106664be4b5aaa330dbb4e43cb7b P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:25.392835 1774 tablet_replica.cc:333] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:25.393030 1774 raft_consensus.cc:2243] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:25.393200 1774 raft_consensus.cc:2272] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:25.321910 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7be9290daabb404f955f05922d897fbf" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:25.394194 1444 leader_election.cc:343] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.326601 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 089c07fc56e64ef4a21e0c0f6879e393 (DEFAULT_TABLE table=default.Table_f8fd3d3a919d4e91844c5ded89bced1b [id=00d5653db09b4574899090f1522b6ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.348054 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4100abc9ff12471e9a4959b6b24f7e99" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:25.394730 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089c07fc56e64ef4a21e0c0f6879e393. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.394976 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5419e516d52e46228d53bc4afcfe4332"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.395586 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5419e516d52e46228d53bc4afcfe4332 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.395980 3496 raft_consensus.cc:493] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.396097 3496 raft_consensus.cc:515] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:25.396760 3496 leader_election.cc:290] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:25.327886 1619 tablet_service.cc:1505] Processing CreateTablet for tablet ec6d23415f28455d804dd770cb1218b3 (DEFAULT_TABLE table=default.Table_3db01f386bf3464cbfee13a63abf924a [id=777eec9a388a4513bd96f34e5c6a5661]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.397851 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec6d23415f28455d804dd770cb1218b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.397810 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0d59c786f9c144b9a5b260a3355773ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.398330 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d59c786f9c144b9a5b260a3355773ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.399590 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d696d859aaad48ceac033959d75bcf11" } new_table_name: "default.Table_b5386eb425a14340919f7bbae554508f" modify_external_catalogs: true
I20251024 19:48:25.360607 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 7145915fe8f94d52868d9b33a7149efa (DEFAULT_TABLE table=default.Table_5be196925fd1419998ee4e85f267f8a4 [id=1a1bcc9018ff4c31b69cd8791fe7e4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.382253 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7be9290daabb404f955f05922d897fbf" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.401796 1616 tablet_service.cc:1505] Processing CreateTablet for tablet 94b7c9b5730840f6af54714fd00219f9 (DEFAULT_TABLE table=default.Table_07d8ae994ade4899b0da421c94480b75 [id=9887d33921574a2d9d9fdef385f1046a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.402163 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b7c9b5730840f6af54714fd00219f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.402163 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7145915fe8f94d52868d9b33a7149efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.405038 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a69c0918d2e4eed9d91b1913ec2c21e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.405656 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 5b224c639bb546aea1fdfd9a0bfc415b (DEFAULT_TABLE table=default.Table_97c3d690de704e908a7ca2e664236bd1 [id=72e109c7131f451c9b9fba54c66a63e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.406044 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b224c639bb546aea1fdfd9a0bfc415b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.408407 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc4e9e3d4725427ea3fc66ec4f183ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.390461 1305 leader_election.cc:304] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.410063 3457 raft_consensus.cc:2749] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:25.411456 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 250f106664be4b5aaa330dbb4e43cb7b (table default.Table_082b885410b34b1cb2f154cdbd9700a8 [id=963d13a1679345489335e30a9aa349b1]) successfully deleted
I20251024 19:48:25.414927 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7145915fe8f94d52868d9b33a7149efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.414870 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 8988137fe63c48bdbe527f223841122b (DEFAULT_TABLE table=default.Table_bd503f203f9943b19088dca94ce8af98 [id=bad903016d2747159195fdcc98f17519]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.415485 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8988137fe63c48bdbe527f223841122b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.416252 1774 ts_tablet_manager.cc:1916] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:25.419020 1447 leader_election.cc:343] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.419168 1447 leader_election.cc:304] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.419466 2858 raft_consensus.cc:2749] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:25.419593 1447 leader_election.cc:343] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.422686 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4100abc9ff12471e9a4959b6b24f7e99
I20251024 19:48:25.422832 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 5ace18d554f14acc892bc956c30e9979
I20251024 19:48:25.423389 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_534425b79d834682916f62eb182e6869" } new_table_name: "default.Table_c0accc5cb0d34a6f833929c12c70af30" modify_external_catalogs: true
I20251024 19:48:25.423418 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_de9a4e8e076649ce9e7aa790a4f1ba7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.423977 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de9a4e8e076649ce9e7aa790a4f1ba7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.422691 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 28f93be6762146398421ac19d2e84e0c
I20251024 19:48:25.426172 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet 4100abc9ff12471e9a4959b6b24f7e99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.426553 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet 28f93be6762146398421ac19d2e84e0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.427166 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ace18d554f14acc892bc956c30e9979 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.429445 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 70dd39a1711a42eeac2da1e394c06d8e (DEFAULT_TABLE table=default.Table_097afacca75e46dfbfec9dd862b21a6c [id=f7e9e8c6c8824f6193f97d2d0bbd0c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.430729 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70dd39a1711a42eeac2da1e394c06d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.431037 1340 tablet_service.cc:1505] Processing CreateTablet for tablet 5b224c639bb546aea1fdfd9a0bfc415b (DEFAULT_TABLE table=default.Table_97c3d690de704e908a7ca2e664236bd1 [id=72e109c7131f451c9b9fba54c66a63e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.431479 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b224c639bb546aea1fdfd9a0bfc415b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.432044 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c86486fc320746d6a92ec7dc81c27180"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.432580 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c86486fc320746d6a92ec7dc81c27180 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.432963 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 70dd39a1711a42eeac2da1e394c06d8e (DEFAULT_TABLE table=default.Table_097afacca75e46dfbfec9dd862b21a6c [id=f7e9e8c6c8824f6193f97d2d0bbd0c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.433321 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70dd39a1711a42eeac2da1e394c06d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.436823 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 977b4070419948d4910e2b69e01ff132
I20251024 19:48:25.436741 1351 tablet_service.cc:1505] Processing CreateTablet for tablet 94b7c9b5730840f6af54714fd00219f9 (DEFAULT_TABLE table=default.Table_07d8ae994ade4899b0da421c94480b75 [id=9887d33921574a2d9d9fdef385f1046a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.437163 1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b7c9b5730840f6af54714fd00219f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.437852 1336 tablet_service.cc:1552] Processing DeleteTablet for tablet 977b4070419948d4910e2b69e01ff132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.438127 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 3b8e8dbca79f4d72a2de51f105b209e4 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:25.438199 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ace18d554f14acc892bc956c30e9979 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.439649 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 70dd39a1711a42eeac2da1e394c06d8e (DEFAULT_TABLE table=default.Table_097afacca75e46dfbfec9dd862b21a6c [id=f7e9e8c6c8824f6193f97d2d0bbd0c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.440078 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70dd39a1711a42eeac2da1e394c06d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.440899 1775 ts_tablet_manager.cc:1929] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:25.441020 1775 log.cc:1199] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/81a3de69a13c459b9b7c02761ab7261e
I20251024 19:48:25.442070 1616 tablet_service.cc:1552] Processing DeleteTablet for tablet 977b4070419948d4910e2b69e01ff132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.445163 1776 log.cc:1199] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/9026d9e801294dd2bfdb9c90e45a0990
I20251024 19:48:25.445420 2858 raft_consensus.cc:493] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.445545 2858 raft_consensus.cc:515] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.419701 1447 leader_election.cc:304] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.446228 1776 ts_tablet_manager.cc:1950] T 9026d9e801294dd2bfdb9c90e45a0990 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:25.447034 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 5b224c639bb546aea1fdfd9a0bfc415b (DEFAULT_TABLE table=default.Table_97c3d690de704e908a7ca2e664236bd1 [id=72e109c7131f451c9b9fba54c66a63e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.447064 1480 tablet_service.cc:1505] Processing CreateTablet for tablet 94b7c9b5730840f6af54714fd00219f9 (DEFAULT_TABLE table=default.Table_07d8ae994ade4899b0da421c94480b75 [id=9887d33921574a2d9d9fdef385f1046a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.447065 3390 raft_consensus.cc:2749] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:25.447414 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b224c639bb546aea1fdfd9a0bfc415b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.447379 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 4100abc9ff12471e9a4959b6b24f7e99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.447849 1766 tablet_bootstrap.cc:492] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:25.447993 1766 ts_tablet_manager.cc:1403] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.613s user 0.014s sys 0.004s
I20251024 19:48:25.447582 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet 28f93be6762146398421ac19d2e84e0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.448791 1775 ts_tablet_manager.cc:1950] T 81a3de69a13c459b9b7c02761ab7261e P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:25.451162 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1fc0f99ca5c42bfb9663e21a7b761c9" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:25.452913 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 28f93be6762146398421ac19d2e84e0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.452965 1774 ts_tablet_manager.cc:1929] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:25.453081 1774 log.cc:1199] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/38818c7997c84cedad5f9b09eef77105
I20251024 19:48:25.453501 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet bec3806757214c869cf1c4cecd7cc26e
W20251024 19:48:25.453810 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1f4bb2e3a5b406fb5b37403667be5d8 consensus state has no leader
I20251024 19:48:25.455325 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet bec3806757214c869cf1c4cecd7cc26e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.455883 1766 raft_consensus.cc:359] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.456107 1766 raft_consensus.cc:385] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:25.456200 1766 raft_consensus.cc:740] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:25.456472 1766 consensus_queue.cc:260] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.457036 1766 ts_tablet_manager.cc:1434] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251024 19:48:25.457341 1766 tablet_bootstrap.cc:492] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:25.458386 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 105169us
I20251024 19:48:25.459111 1766 tablet_bootstrap.cc:654] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:25.460181 1481 tablet_service.cc:1552] Processing DeleteTablet for tablet 4100abc9ff12471e9a4959b6b24f7e99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.460238 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1023) took 5242 ms. Trace:
I20251024 19:48:25.460274 2858 leader_election.cc:290] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:25.460453 1444 leader_election.cc:343] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.460521 3387 raft_consensus.cc:493] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.460633 3387 raft_consensus.cc:515] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.461187 3387 leader_election.cc:290] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.460368 1775 rpcz_store.cc:276] 1024 19:48:20.206253 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.213126 (+ 6873us) service_pool.cc:225] Handling call
1024 19:48:25.124612 (+4911486us) ts_tablet_manager.cc:1151] Deleting tablet 81a3de69a13c459b9b7c02761ab7261e
1024 19:48:25.124623 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.275743 (+151120us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:25.440787 (+165044us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:25.449065 (+ 8278us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1011,"tablet-delete.queue_time_us":4911104}
I20251024 19:48:25.461753 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet 977b4070419948d4910e2b69e01ff132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.461810 1775 tablet_replica.cc:333] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:25.461970 1478 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ace18d554f14acc892bc956c30e9979 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.462169 1775 raft_consensus.cc:2243] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:25.462205 1586 leader_election.cc:343] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.462406 1775 raft_consensus.cc:2272] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:25.463063 1775 ts_tablet_manager.cc:1916] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:25.467451 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet b632acc1348e49f8a1eea019c007780c (table default.Table_361840e201d943c18014ec71a255cc9c [id=dc00ee30038e49fb9f86499cfb985a0e]). Expected version 2 got 0
I20251024 19:48:25.469223 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "769f576810ab49368eb0fc7837b7da58" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:25.469461 1505 raft_consensus.cc:2468] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:25.470115 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1d26e02ad2945b39cf27738ed5b7cf4" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:25.470311 1511 raft_consensus.cc:2468] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:25.470551 1767 tablet_bootstrap.cc:654] T ccd46da86fe444baa675d3391cdd410c P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:25.472723 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1056) took 4249 ms. Trace:
I20251024 19:48:25.472820 1776 rpcz_store.cc:276] 1024 19:48:21.222877 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.225512 (+ 2635us) service_pool.cc:225] Handling call
1024 19:48:25.082646 (+3857134us) ts_tablet_manager.cc:1151] Deleting tablet 9026d9e801294dd2bfdb9c90e45a0990
1024 19:48:25.082657 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.239592 (+156935us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:25.304956 (+ 65364us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:25.472703 (+167747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3856295}
I20251024 19:48:25.473176 1776 tablet_replica.cc:333] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:25.473469 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 9026d9e801294dd2bfdb9c90e45a0990 (table default.Table_9e14814a2db64e0b9602e41f926f3b62 [id=1841054cd03e4989a236ebf134fa7755]) successfully deleted
I20251024 19:48:25.476038 1190 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 1999b751e9344dcf81d26226c0589c8f (table default.Table_9147a98b3bf9483ea7b7a6f17130d4fc [id=d743705c21da4b89abedf2eb34ddbe09]). Expected version 1 got 0
I20251024 19:48:25.476634 1586 leader_election.cc:304] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:25.477023 3387 raft_consensus.cc:2804] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:25.477052 1586 leader_election.cc:304] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:25.477101 3387 raft_consensus.cc:493] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:25.477160 3387 raft_consensus.cc:3060] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.477633 3360 raft_consensus.cc:2804] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:25.477725 3360 raft_consensus.cc:493] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:25.477779 3360 raft_consensus.cc:3060] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.479647 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 81a3de69a13c459b9b7c02761ab7261e (table default.Table_b5a66c4e9cd8408782690b66cdb87473 [id=3d82ad2f9e184043aed4ec0b05df724f]) successfully deleted
I20251024 19:48:25.480106 3387 raft_consensus.cc:515] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:25.480293 3360 raft_consensus.cc:515] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.480276 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1f4d97aa9d894d43875a410601cc63b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:25.480691 3387 leader_election.cc:290] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:25.480742 3360 leader_election.cc:290] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:25.480852 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f4d97aa9d894d43875a410601cc63b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.481225 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 2f068c3ae13243888ceb972d31957bcb (DEFAULT_TABLE table=default.Table_54c5eb4772114f2e829fe31c91a9292d [id=9945053de4384295928dd7e221297394]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.481817 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 8988137fe63c48bdbe527f223841122b (DEFAULT_TABLE table=default.Table_bd503f203f9943b19088dca94ce8af98 [id=bad903016d2747159195fdcc98f17519]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.482251 2858 raft_consensus.cc:493] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.482456 2858 raft_consensus.cc:515] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.482807 1478 tablet_service.cc:1505] Processing CreateTablet for tablet 8988137fe63c48bdbe527f223841122b (DEFAULT_TABLE table=default.Table_bd503f203f9943b19088dca94ce8af98 [id=bad903016d2747159195fdcc98f17519]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.483132 2858 leader_election.cc:290] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.483247 1478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8988137fe63c48bdbe527f223841122b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.453987 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1fc0f99ca5c42bfb9663e21a7b761c9" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.455176 1768 tablet_bootstrap.cc:492] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:25.466678 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "769f576810ab49368eb0fc7837b7da58" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.487848 1768 ts_tablet_manager.cc:1403] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.418s user 0.017s sys 0.008s
I20251024 19:48:25.481853 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 2f068c3ae13243888ceb972d31957bcb (DEFAULT_TABLE table=default.Table_54c5eb4772114f2e829fe31c91a9292d [id=9945053de4384295928dd7e221297394]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.486340 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b2992a738b440d798f5e61cc45ec088" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.488711 1768 raft_consensus.cc:359] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.488885 1768 raft_consensus.cc:385] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:25.488950 1768 raft_consensus.cc:740] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:25.488933 1776 raft_consensus.cc:2243] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:25.489125 1768 consensus_queue.cc:260] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.489694 1768 ts_tablet_manager.cc:1434] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:25.489979 1768 tablet_bootstrap.cc:492] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:25.490991 1774 ts_tablet_manager.cc:1950] T 38818c7997c84cedad5f9b09eef77105 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:25.492462 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 7785331cd8244ed097cb33c4575167ce (DEFAULT_TABLE table=default.Table_dcdcda89edd64b598f444eb9f4d98048 [id=97de44563cea476fa3682d0fa955be97]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:25.493211 1447 leader_election.cc:343] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.493369 1447 leader_election.cc:304] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.493762 2858 raft_consensus.cc:2749] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:25.493908 1447 leader_election.cc:343] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.494458 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b7d69816c17a4e9cb68d17da3226cc02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.495057 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7d69816c17a4e9cb68d17da3226cc02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.504647 1774 tablet_replica.cc:333] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:25.504781 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1d26e02ad2945b39cf27738ed5b7cf4" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.505076 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f068c3ae13243888ceb972d31957bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.509825 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 70a0f4e026ca4831adb8b916fc5fc53b (DEFAULT_TABLE table=default.Table_02497c364ee24472a2d1ca697b81e22e [id=6c5e8e877ad84711a1d47168eac5e731]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.510344 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 6942b876b722478aa9e4d7a50570b597 (DEFAULT_TABLE table=default.Table_06e859c14ac948dd8996f7fbec456138 [id=b07cd575087c435987e1d97d8ca6db94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.510637 1616 tablet_service.cc:1505] Processing CreateTablet for tablet d0f6afbee53d4b7d94105cb1487e3fd8 (DEFAULT_TABLE table=default.Table_7d7b19adc8404444ab1e322bac3adf0c [id=cc61ce4546794e57bc0d2d93e4c2f8de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.511354 1776 raft_consensus.cc:2272] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:25.512346 1776 ts_tablet_manager.cc:1916] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:25.512914 1768 tablet_bootstrap.cc:654] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:25.516180 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4bd7d9d519c243d7acc4434dd39b18a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.516765 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bd7d9d519c243d7acc4434dd39b18a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:25.518420 1586 leader_election.cc:343] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:25.518896 1586 leader_election.cc:343] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.518950 1491 tablet_service.cc:1552] Processing DeleteTablet for tablet bec3806757214c869cf1c4cecd7cc26e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.520371 3496 consensus_queue.cc:1048] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:25.520565 1774 raft_consensus.cc:2243] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:25.520779 1774 raft_consensus.cc:2272] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:25.522291 1510 raft_consensus.cc:2468] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:25.526700 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 7785331cd8244ed097cb33c4575167ce (DEFAULT_TABLE table=default.Table_dcdcda89edd64b598f444eb9f4d98048 [id=97de44563cea476fa3682d0fa955be97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.527932 1342 tablet_service.cc:1505] Processing CreateTablet for tablet d0f6afbee53d4b7d94105cb1487e3fd8 (DEFAULT_TABLE table=default.Table_7d7b19adc8404444ab1e322bac3adf0c [id=cc61ce4546794e57bc0d2d93e4c2f8de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.528215 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet bec3806757214c869cf1c4cecd7cc26e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.528716 2858 raft_consensus.cc:493] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.528918 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "769f576810ab49368eb0fc7837b7da58" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:25.529253 1586 leader_election.cc:304] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:25.530094 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 2f068c3ae13243888ceb972d31957bcb (DEFAULT_TABLE table=default.Table_54c5eb4772114f2e829fe31c91a9292d [id=9945053de4384295928dd7e221297394]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.530995 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b7c9b5730840f6af54714fd00219f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.531006 1774 ts_tablet_manager.cc:1916] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:25.531520 1767 tablet_bootstrap.cc:492] T ccd46da86fe444baa675d3391cdd410c P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:25.531826 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 6942b876b722478aa9e4d7a50570b597 (DEFAULT_TABLE table=default.Table_06e859c14ac948dd8996f7fbec456138 [id=b07cd575087c435987e1d97d8ca6db94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.532073 1767 ts_tablet_manager.cc:1403] T ccd46da86fe444baa675d3391cdd410c P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.258s user 0.001s sys 0.003s
I20251024 19:48:25.533084 1349 tablet_service.cc:1505] Processing CreateTablet for tablet 398e73d6d576487e968c9083175965aa (DEFAULT_TABLE table=default.Table_40837137d2a145828e6dcbd42e88e393 [id=a174648aa0f944d193fde7710520a5e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.533080 1767 raft_consensus.cc:359] T ccd46da86fe444baa675d3391cdd410c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.533396 1767 raft_consensus.cc:385] T ccd46da86fe444baa675d3391cdd410c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:25.533505 1767 raft_consensus.cc:740] T ccd46da86fe444baa675d3391cdd410c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:25.533851 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "769f576810ab49368eb0fc7837b7da58" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:25.534180 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1d26e02ad2945b39cf27738ed5b7cf4" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:25.536191 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 45cff9bcc4924e0081518c284f074940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.536437 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 398e73d6d576487e968c9083175965aa (DEFAULT_TABLE table=default.Table_40837137d2a145828e6dcbd42e88e393 [id=a174648aa0f944d193fde7710520a5e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.536630 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 38818c7997c84cedad5f9b09eef77105 (table default.Table_d7a9ef5436984a1b83dbeac77c302549 [id=4c4868b6a38046faaf24f65fd00420cb]) successfully deleted
I20251024 19:48:25.537192 3496 raft_consensus.cc:2804] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:25.537729 3496 raft_consensus.cc:697] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:25.538137 3496 consensus_queue.cc:237] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.539052 1768 tablet_bootstrap.cc:492] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:25.539294 1768 ts_tablet_manager.cc:1403] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20251024 19:48:25.540920 1768 raft_consensus.cc:359] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.541241 1768 raft_consensus.cc:385] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:25.541391 1768 raft_consensus.cc:740] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:25.541764 1768 consensus_queue.cc:260] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.544570 3390 consensus_queue.cc:1048] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:25.545394 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b2992a738b440d798f5e61cc45ec088" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:25.551668 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_323cd4e217904fa5a3671990e7b67ce9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.552340 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_323cd4e217904fa5a3671990e7b67ce9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:25.555979 1444 leader_election.cc:343] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.558131 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f068c3ae13243888ceb972d31957bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.560876 1768 ts_tablet_manager.cc:1434] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.021s user 0.002s sys 0.001s
I20251024 19:48:25.561854 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7785331cd8244ed097cb33c4575167ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.566054 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f6afbee53d4b7d94105cb1487e3fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.569766 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a0f4e026ca4831adb8b916fc5fc53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.573735 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6942b876b722478aa9e4d7a50570b597. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.574035 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 398e73d6d576487e968c9083175965aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.577294 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7785331cd8244ed097cb33c4575167ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.577463 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6942b876b722478aa9e4d7a50570b597. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.579216 1183 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet b632acc1348e49f8a1eea019c007780c (table default.Table_361840e201d943c18014ec71a255cc9c [id=dc00ee30038e49fb9f86499cfb985a0e]). Expected version 2 got 0
I20251024 19:48:25.581180 1617 tablet_service.cc:1505] Processing CreateTablet for tablet f4c6bc454f1a4ffb975c3cd6ef9a491b (DEFAULT_TABLE table=default.Table_dd7cc9b0637441719717ddd3eeafb570 [id=52e686513d0f4c9fabdfe9eb51aa5f31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.581630 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4c6bc454f1a4ffb975c3cd6ef9a491b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.581677 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f6afbee53d4b7d94105cb1487e3fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.585986 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 1d5bb186d0d74c8c8d135e04783d3a7a (DEFAULT_TABLE table=default.Table_46ccc3bbcdad400884c16fd5425e5c3c [id=66a867d40e3f42c9a44d6e19ffeacfe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.573738 1767 consensus_queue.cc:260] T ccd46da86fe444baa675d3391cdd410c P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.598655 1767 ts_tablet_manager.cc:1434] T ccd46da86fe444baa675d3391cdd410c P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.066s user 0.001s sys 0.001s
I20251024 19:48:25.599315 1183 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 1999b751e9344dcf81d26226c0589c8f (table default.Table_9147a98b3bf9483ea7b7a6f17130d4fc [id=d743705c21da4b89abedf2eb34ddbe09]). Expected version 1 got 0
I20251024 19:48:25.599749 1767 tablet_bootstrap.cc:492] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:25.601135 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d5bb186d0d74c8c8d135e04783d3a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.601876 1768 tablet_bootstrap.cc:492] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:25.604473 1768 tablet_bootstrap.cc:654] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:25.606827 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 70a0f4e026ca4831adb8b916fc5fc53b (DEFAULT_TABLE table=default.Table_02497c364ee24472a2d1ca697b81e22e [id=6c5e8e877ad84711a1d47168eac5e731]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.607288 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a0f4e026ca4831adb8b916fc5fc53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.610651 1768 tablet_bootstrap.cc:492] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
W20251024 19:48:25.611004 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ba845f3f8984585af370c3d39f0ebb9 consensus state has no leader
I20251024 19:48:25.611006 1768 ts_tablet_manager.cc:1403] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20251024 19:48:25.612473 1768 raft_consensus.cc:359] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.612876 1768 raft_consensus.cc:385] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:25.613085 1768 raft_consensus.cc:740] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:25.613519 1768 consensus_queue.cc:260] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.614533 1627 tablet_service.cc:1552] Processing DeleteTablet for tablet 769f576810ab49368eb0fc7837b7da58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.610707 1774 ts_tablet_manager.cc:1929] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:25.616082 1774 log.cc:1199] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/2e7b3288182744969e217d4f174fe4ea
I20251024 19:48:25.616144 1768 ts_tablet_manager.cc:1434] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:48:25.617017 1774 ts_tablet_manager.cc:1950] T 2e7b3288182744969e217d4f174fe4ea P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:25.618361 1768 tablet_bootstrap.cc:492] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:25.629201 1183 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet b632acc1348e49f8a1eea019c007780c (table default.Table_361840e201d943c18014ec71a255cc9c [id=dc00ee30038e49fb9f86499cfb985a0e]). Expected version 2 got 0
I20251024 19:48:25.630986 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1d26e02ad2945b39cf27738ed5b7cf4" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:25.642051 1766 tablet_bootstrap.cc:492] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:25.643256 1627 tablet_service.cc:1505] Processing CreateTablet for tablet e05eb2fb1ba742629c3e89f96630e986 (DEFAULT_TABLE table=default.Table_4f639b4e1ca94db3a70d5ce8a2d5b050 [id=f568303a5da64fb49b3adbf081eb898d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.643707 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e05eb2fb1ba742629c3e89f96630e986. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:25.644518 1586 leader_election.cc:343] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:25.644783 1586 leader_election.cc:343] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.645071 1509 raft_consensus.cc:3060] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.647684 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 0f6f8cb6415d490184383c6f1ebe99f8 (DEFAULT_TABLE table=default.Table_1fcb39e601194b41bfaaa87ec866ae47 [id=7bdd81d3c7e64a1d91c64a156c9e63ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.648043 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f6f8cb6415d490184383c6f1ebe99f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.649885 1617 tablet_service.cc:1505] Processing CreateTablet for tablet c910a21a05ea4d3a8d8a400a3cc58f2e (DEFAULT_TABLE table=default.Table_5fa0890a968b4ba785c99c435ffded4a [id=4cb969aac493462dabbeb714e1cdc1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.650229 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c910a21a05ea4d3a8d8a400a3cc58f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.650449 1767 tablet_bootstrap.cc:654] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:25.644507 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 6942b876b722478aa9e4d7a50570b597 (DEFAULT_TABLE table=default.Table_06e859c14ac948dd8996f7fbec456138 [id=b07cd575087c435987e1d97d8ca6db94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.660604 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 1d5bb186d0d74c8c8d135e04783d3a7a (DEFAULT_TABLE table=default.Table_46ccc3bbcdad400884c16fd5425e5c3c [id=66a867d40e3f42c9a44d6e19ffeacfe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.661089 1338 tablet_service.cc:1505] Processing CreateTablet for tablet f4c6bc454f1a4ffb975c3cd6ef9a491b (DEFAULT_TABLE table=default.Table_dd7cc9b0637441719717ddd3eeafb570 [id=52e686513d0f4c9fabdfe9eb51aa5f31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.661424 1342 tablet_service.cc:1505] Processing CreateTablet for tablet c910a21a05ea4d3a8d8a400a3cc58f2e (DEFAULT_TABLE table=default.Table_5fa0890a968b4ba785c99c435ffded4a [id=4cb969aac493462dabbeb714e1cdc1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.661891 1349 tablet_service.cc:1505] Processing CreateTablet for tablet e05eb2fb1ba742629c3e89f96630e986 (DEFAULT_TABLE table=default.Table_4f639b4e1ca94db3a70d5ce8a2d5b050 [id=f568303a5da64fb49b3adbf081eb898d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.664402 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8988137fe63c48bdbe527f223841122b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.668278 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_845cc413de744780ad82fabf47646b46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.668845 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_845cc413de744780ad82fabf47646b46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.670923 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9ebd164c5e924a7eaff697af91c4c1a5" } new_table_name: "default.Table_c5519e755b3c4e62a7c4f70457f3c3c6" modify_external_catalogs: true
I20251024 19:48:25.681227 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_159163726a8743d6a8e6ee5e5d68c082" } modify_external_catalogs: true
I20251024 19:48:25.681442 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_159163726a8743d6a8e6ee5e5d68c082" } modify_external_catalogs: true
I20251024 19:48:25.684115 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d92ce2f682b547aca4675850f59c89bc" } new_table_name: "default.Table_9a0a6de853064737a16d93efb4aeb8ce" modify_external_catalogs: true
I20251024 19:48:25.687852 1625 tablet_service.cc:1505] Processing CreateTablet for tablet ca38f3a765264dcc83f73a5d586cfea9 (DEFAULT_TABLE table=default.Table_92ca5409f01446e494834187689f234d [id=212636f6527a4647a129d04c6126cdc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.691021 2858 raft_consensus.cc:515] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.691787 2858 leader_election.cc:290] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.691962 1511 raft_consensus.cc:3060] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:25.695222 1511 raft_consensus.cc:2468] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:25.695500 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f068c3ae13243888ceb972d31957bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.700659 1775 ts_tablet_manager.cc:1929] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:25.700811 1775 log.cc:1199] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/b20a6845ae1645c0b51f096d471d4ccc
I20251024 19:48:25.701190 1775 ts_tablet_manager.cc:1950] T b20a6845ae1645c0b51f096d471d4ccc P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:25.701524 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1025) took 5297 ms. Trace:
I20251024 19:48:25.701651 1775 rpcz_store.cc:276] 1024 19:48:20.404030 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.404148 (+ 118us) service_pool.cc:225] Handling call
1024 19:48:25.461708 (+5057560us) ts_tablet_manager.cc:1151] Deleting tablet b20a6845ae1645c0b51f096d471d4ccc
1024 19:48:25.461722 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.466731 (+ 5009us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:25.533130 (+ 66399us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:25.701491 (+168361us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5057309}
I20251024 19:48:25.701920 1444 leader_election.cc:304] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.702061 1775 tablet_replica.cc:333] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:25.702296 1775 raft_consensus.cc:2243] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:25.702320 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 154537us
I20251024 19:48:25.702509 1775 raft_consensus.cc:2272] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:25.704772 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6942b876b722478aa9e4d7a50570b597. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.704993 2858 raft_consensus.cc:2749] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:25.713472 3457 raft_consensus.cc:493] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.713685 3457 raft_consensus.cc:515] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.714264 3457 leader_election.cc:290] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:25.714975 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1d26e02ad2945b39cf27738ed5b7cf4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:25.716645 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 70a0f4e026ca4831adb8b916fc5fc53b (DEFAULT_TABLE table=default.Table_02497c364ee24472a2d1ca697b81e22e [id=6c5e8e877ad84711a1d47168eac5e731]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.717154 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a0f4e026ca4831adb8b916fc5fc53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.720296 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a99c55b564b946b6a76c2b981640cdcd" } modify_external_catalogs: true
I20251024 19:48:25.720479 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a99c55b564b946b6a76c2b981640cdcd" } modify_external_catalogs: true
I20251024 19:48:25.721689 1487 tablet_service.cc:1505] Processing CreateTablet for tablet d0f6afbee53d4b7d94105cb1487e3fd8 (DEFAULT_TABLE table=default.Table_7d7b19adc8404444ab1e322bac3adf0c [id=cc61ce4546794e57bc0d2d93e4c2f8de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.722132 1487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f6afbee53d4b7d94105cb1487e3fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.722277 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d61970c7cdd9475d9d6da8dc5f80a232" } modify_external_catalogs: true
I20251024 19:48:25.722424 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d61970c7cdd9475d9d6da8dc5f80a232" } modify_external_catalogs: true
I20251024 19:48:25.723800 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_09f15fac93da45ab80457bd1efc49464" } modify_external_catalogs: true
I20251024 19:48:25.723937 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_09f15fac93da45ab80457bd1efc49464" } modify_external_catalogs: true
I20251024 19:48:25.725170 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5afc602c5d154cf48565fa17d88d7c68" } modify_external_catalogs: true
I20251024 19:48:25.725297 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5afc602c5d154cf48565fa17d88d7c68" } modify_external_catalogs: true
I20251024 19:48:25.725620 1766 ts_tablet_manager.cc:1403] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.185s user 0.005s sys 0.013s
I20251024 19:48:25.726374 1766 raft_consensus.cc:359] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:25.726531 1766 raft_consensus.cc:385] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:25.726579 1766 raft_consensus.cc:740] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:25.726787 1766 consensus_queue.cc:260] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:25.727264 1766 ts_tablet_manager.cc:1434] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:25.727540 1766 tablet_bootstrap.cc:492] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:25.727468 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6ef65af7e15c4360b7808f7dd020727b" } new_table_name: "default.Table_1c68eae2ab444563993e0a40d23ce326" modify_external_catalogs: true
I20251024 19:48:25.728627 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet bc61fded84b24e00b299e5d342076708 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:25.729081 1766 tablet_bootstrap.cc:654] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:25.731669 1775 ts_tablet_manager.cc:1916] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:25.736413 3455 raft_consensus.cc:493] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.736567 3455 raft_consensus.cc:515] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.736757 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_311679c71ec1481e893a01440c1e07ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:25.737195 3455 leader_election.cc:290] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:25.737313 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_311679c71ec1481e893a01440c1e07ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.737321 1509 raft_consensus.cc:2468] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:25.737671 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b22b58404b8b441ba97ef536885a66e0" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:25.738127 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 7785331cd8244ed097cb33c4575167ce (DEFAULT_TABLE table=default.Table_dcdcda89edd64b598f444eb9f4d98048 [id=97de44563cea476fa3682d0fa955be97]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:25.738193 1444 leader_election.cc:343] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:25.742810 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7ffb227901f49b99fb57c5c6e138c8c consensus state has no leader
I20251024 19:48:25.743204 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet c084db163d874639b4385f41e9efaa18
W20251024 19:48:25.744251 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96e8588e092f45c2b10f4e1e97408c47 consensus state has no leader
I20251024 19:48:25.752434 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet d72f4761820e430cb1a97d83cd30c75d
I20251024 19:48:25.753147 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3c608ac55e9e42bc9c9b27beec717e30
I20251024 19:48:25.753732 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b383a83425654723bd1d9c3a39b187be
I20251024 19:48:25.754266 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet e0228d5b788e4c42bf4ecce876ecc3f4
I20251024 19:48:25.754709 1766 tablet_bootstrap.cc:492] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:25.754850 1766 ts_tablet_manager.cc:1403] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
W20251024 19:48:25.754949 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 529580599bfa4800973b8540ab13d22a consensus state has no leader
I20251024 19:48:25.755656 1766 raft_consensus.cc:359] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.755826 1766 raft_consensus.cc:385] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:25.755879 1766 raft_consensus.cc:740] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:25.756090 1766 consensus_queue.cc:260] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.756501 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet c084db163d874639b4385f41e9efaa18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.756603 1766 ts_tablet_manager.cc:1434] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:25.756906 1766 tablet_bootstrap.cc:492] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:25.757691 1477 tablet_service.cc:1505] Processing CreateTablet for tablet f4c6bc454f1a4ffb975c3cd6ef9a491b (DEFAULT_TABLE table=default.Table_dd7cc9b0637441719717ddd3eeafb570 [id=52e686513d0f4c9fabdfe9eb51aa5f31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.758375 1766 tablet_bootstrap.cc:654] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:25.758560 1482 tablet_service.cc:1505] Processing CreateTablet for tablet c910a21a05ea4d3a8d8a400a3cc58f2e (DEFAULT_TABLE table=default.Table_5fa0890a968b4ba785c99c435ffded4a [id=4cb969aac493462dabbeb714e1cdc1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.758991 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 5d62c5c336e14c65bf6734c66e9be198 (DEFAULT_TABLE table=default.Table_32612c9da3dd4bce8d35d2fdc3bb4bd4 [id=796ec6511bc64adeac0a19b4381b73ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.704545 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64499c8734d34c66b0b52a2cd3ef0673" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:25.761780 1628 tablet_service.cc:1505] Processing CreateTablet for tablet b2a273b68eee4caeb468326187d6600c (DEFAULT_TABLE table=default.Table_50a7c4f2e9b94c2f9781f366cbac5b92 [id=fb47f2546fdc43e79bbbb9d581293c39]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:25.762229 1444 leader_election.cc:343] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.763461 3141 consensus_queue.cc:1048] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:25.764017 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 1d5bb186d0d74c8c8d135e04783d3a7a (DEFAULT_TABLE table=default.Table_46ccc3bbcdad400884c16fd5425e5c3c [id=66a867d40e3f42c9a44d6e19ffeacfe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.764647 1187 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 1999b751e9344dcf81d26226c0589c8f (table default.Table_9147a98b3bf9483ea7b7a6f17130d4fc [id=d743705c21da4b89abedf2eb34ddbe09]). Expected version 1 got 0
I20251024 19:48:25.764931 1187 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet f965111ac60841dbb9e6ecc21ff3fec0 (table default.Table_b5386eb425a14340919f7bbae554508f [id=c727179cd01648cd86e1d9eca2396cae]). Expected version 1 got 0
I20251024 19:48:25.765229 1187 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet a1fbbba88a294adfbf97c0343d86698e (table default.Table_7bc8cd4a930b44dfafa473297875594d [id=a60c8222aa0248c091fab8dc34100847]). Expected version 2 got 0
I20251024 19:48:25.766381 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet b20a6845ae1645c0b51f096d471d4ccc (table default.Table_50e12cf5f39d4fa2983835ef0dbbc3d6 [id=201e5232763a4940a1288a56a695f71c]) successfully deleted
I20251024 19:48:25.768431 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c910a21a05ea4d3a8d8a400a3cc58f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.768961 1477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4c6bc454f1a4ffb975c3cd6ef9a491b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.769637 1491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7785331cd8244ed097cb33c4575167ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.769611 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet d72f4761820e430cb1a97d83cd30c75d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.769737 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 83bb9b2dbf5c48f38ed8b1f29d1ba466 (DEFAULT_TABLE table=default.Table_5046f8911611433295fd660877c0c7e3 [id=6d4415fcba5d41d5b77acd4c14e0ff78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.769944 1621 tablet_service.cc:1552] Processing DeleteTablet for tablet b383a83425654723bd1d9c3a39b187be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.770089 1626 tablet_service.cc:1552] Processing DeleteTablet for tablet e0228d5b788e4c42bf4ecce876ecc3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.770288 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d5bb186d0d74c8c8d135e04783d3a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.763073 1586 leader_election.cc:304] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:25.771229 1586 leader_election.cc:304] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:25.770798 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca38f3a765264dcc83f73a5d586cfea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.772524 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d62c5c336e14c65bf6734c66e9be198. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.772716 3497 raft_consensus.cc:2804] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:25.772755 1776 ts_tablet_manager.cc:1929] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:25.772836 3497 raft_consensus.cc:697] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:25.772897 3387 raft_consensus.cc:2804] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:25.773084 3387 raft_consensus.cc:697] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:25.772991 1776 log.cc:1199] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/28005301f9de48e5826f46c79aeb5fe5
I20251024 19:48:25.773989 1776 ts_tablet_manager.cc:1950] T 28005301f9de48e5826f46c79aeb5fe5 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:25.774390 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1055) took 4551 ms. Trace:
I20251024 19:48:25.774544 1776 rpcz_store.cc:276] 1024 19:48:21.222740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.226075 (+ 3335us) service_pool.cc:225] Handling call
1024 19:48:25.473127 (+4247052us) ts_tablet_manager.cc:1151] Deleting tablet 28005301f9de48e5826f46c79aeb5fe5
1024 19:48:25.473136 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.628940 (+155804us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:25.772653 (+143713us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:25.774369 (+ 1716us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15371,"tablet-delete.queue_time_us":4246613}
I20251024 19:48:25.775090 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 28005301f9de48e5826f46c79aeb5fe5 (table default.Table_e74966425ce8408da939910bc645012d [id=e345af66b89e48f9afe96000ce50179c]) successfully deleted
I20251024 19:48:25.775710 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1062) took 4212 ms. Trace:
I20251024 19:48:25.769846 1615 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c608ac55e9e42bc9c9b27beec717e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:25.776714 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a273b68eee4caeb468326187d6600c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.776901 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 398e73d6d576487e968c9083175965aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.777055 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83bb9b2dbf5c48f38ed8b1f29d1ba466. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.780187 3497 consensus_queue.cc:237] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:25.773305 3387 consensus_queue.cc:237] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.776348 1776 rpcz_store.cc:276] 1024 19:48:21.563754 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.564052 (+ 298us) service_pool.cc:225] Handling call
1024 19:48:25.775559 (+4211507us) ts_tablet_manager.cc:1151] Deleting tablet 703441e1af8e4356afd22ef68dac87a2
1024 19:48:25.775570 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.775692 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4211137}
W20251024 19:48:25.786883 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 703441e1af8e4356afd22ef68dac87a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 703441e1af8e4356afd22ef68dac87a2 already in progress: creating tablet
I20251024 19:48:25.787364 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1068) took 4138 ms. Trace:
I20251024 19:48:25.787541 1776 rpcz_store.cc:276] 1024 19:48:21.648675 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.648910 (+ 235us) service_pool.cc:225] Handling call
1024 19:48:25.787234 (+4138324us) ts_tablet_manager.cc:1151] Deleting tablet 68d85226ac5b4e4ab22a49dc419e375a
1024 19:48:25.787246 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.787345 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4138145}
W20251024 19:48:25.789638 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 68d85226ac5b4e4ab22a49dc419e375a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68d85226ac5b4e4ab22a49dc419e375a already in progress: creating tablet
I20251024 19:48:25.790131 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1072) took 4047 ms. Trace:
I20251024 19:48:25.790269 1776 rpcz_store.cc:276] 1024 19:48:21.743010 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.745305 (+ 2295us) service_pool.cc:225] Handling call
1024 19:48:25.789991 (+4044686us) ts_tablet_manager.cc:1151] Deleting tablet 4ddce416119e4f08b81be4b5e6760144
1024 19:48:25.790005 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.790111 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4044432}
W20251024 19:48:25.790714 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4ddce416119e4f08b81be4b5e6760144 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ddce416119e4f08b81be4b5e6760144 already in progress: creating tablet
I20251024 19:48:25.791083 1776 tablet_replica.cc:333] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:25.791311 1776 raft_consensus.cc:2243] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:25.791603 1776 raft_consensus.cc:2272] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:25.791805 1474 tablet_service.cc:1505] Processing CreateTablet for tablet e05eb2fb1ba742629c3e89f96630e986 (DEFAULT_TABLE table=default.Table_4f639b4e1ca94db3a70d5ce8a2d5b050 [id=f568303a5da64fb49b3adbf081eb898d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.792249 1474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e05eb2fb1ba742629c3e89f96630e986. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.792377 1776 ts_tablet_manager.cc:1916] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:25.798431 1472 tablet_service.cc:1505] Processing CreateTablet for tablet 0f6f8cb6415d490184383c6f1ebe99f8 (DEFAULT_TABLE table=default.Table_1fcb39e601194b41bfaaa87ec866ae47 [id=7bdd81d3c7e64a1d91c64a156c9e63ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.798833 1472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f6f8cb6415d490184383c6f1ebe99f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.803261 1776 ts_tablet_manager.cc:1929] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:25.803386 1776 log.cc:1199] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/edb49884389c4dba957f0ee5af4ae0e4
I20251024 19:48:25.803728 1776 ts_tablet_manager.cc:1950] T edb49884389c4dba957f0ee5af4ae0e4 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:25.818406 3456 raft_consensus.cc:493] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af3952de050140d383f6b0de7666fa97)
I20251024 19:48:25.818586 3456 raft_consensus.cc:515] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:25.819180 3456 leader_election.cc:290] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:25.819473 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2671108031db449cae2135835fc5f739 (table default.Table_d3ffdf5cbfd84551b1e314d4a772be1a [id=2f6b7b685af84337bd64141ef2a9e992], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2671108031db449cae2135835fc5f739 consensus state has no leader
I20251024 19:48:25.823668 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1073) took 4067 ms. Trace:
I20251024 19:48:25.823832 1776 rpcz_store.cc:276] 1024 19:48:21.756464 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.765633 (+ 9169us) service_pool.cc:225] Handling call
1024 19:48:25.791034 (+4025401us) ts_tablet_manager.cc:1151] Deleting tablet edb49884389c4dba957f0ee5af4ae0e4
1024 19:48:25.791044 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.799795 (+ 8751us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:25.803174 (+ 3379us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:25.823644 (+ 20470us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19523,"tablet-delete.queue_time_us":4025236}
I20251024 19:48:25.824303 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet edb49884389c4dba957f0ee5af4ae0e4 (table default.Table_07aae1757a3444a4bbb8d729d6f08a1d [id=b410120363fe4ba5907be2c8793b5e8c]) successfully deleted
I20251024 19:48:25.824767 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1075) took 4013 ms. Trace:
I20251024 19:48:25.824884 1776 rpcz_store.cc:276] 1024 19:48:21.811129 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.811195 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:25.824630 (+4013435us) ts_tablet_manager.cc:1151] Deleting tablet c89fa8932dbb4f4fbf7fd9f57bff27c3
1024 19:48:25.824640 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.824749 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4013112}
W20251024 19:48:25.825503 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet c89fa8932dbb4f4fbf7fd9f57bff27c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c89fa8932dbb4f4fbf7fd9f57bff27c3 already in progress: creating tablet
I20251024 19:48:25.825892 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1077) took 3748 ms. Trace:
I20251024 19:48:25.826310 1776 rpcz_store.cc:276] 1024 19:48:22.077034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.077139 (+ 105us) service_pool.cc:225] Handling call
1024 19:48:25.825768 (+3748629us) ts_tablet_manager.cc:1151] Deleting tablet aa08555dbe1e4ad8968c534510eca9ff
1024 19:48:25.825780 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.825874 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3747853}
W20251024 19:48:25.827390 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet aa08555dbe1e4ad8968c534510eca9ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa08555dbe1e4ad8968c534510eca9ff already in progress: creating tablet
I20251024 19:48:25.829262 1775 ts_tablet_manager.cc:1929] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:25.829464 1775 log.cc:1199] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/0266cef4e2034303978d7456c47ea98c
I20251024 19:48:25.836089 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1078) took 3652 ms. Trace:
I20251024 19:48:25.836198 1776 rpcz_store.cc:276] 1024 19:48:22.183199 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.183249 (+ 50us) service_pool.cc:225] Handling call
1024 19:48:25.827225 (+3643976us) ts_tablet_manager.cc:1151] Deleting tablet ce374bf694604250b3bbef15856ad01d
1024 19:48:25.835926 (+ 8701us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.836063 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3643802}
W20251024 19:48:25.836637 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet ce374bf694604250b3bbef15856ad01d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce374bf694604250b3bbef15856ad01d already in progress: creating tablet
I20251024 19:48:25.837822 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1079) took 3621 ms. Trace:
I20251024 19:48:25.837937 1776 rpcz_store.cc:276] 1024 19:48:22.216224 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.216288 (+ 64us) service_pool.cc:225] Handling call
1024 19:48:25.837668 (+3621380us) ts_tablet_manager.cc:1151] Deleting tablet d9367428428648f7ae07722ea09d70a7
1024 19:48:25.837683 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.837797 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3621180}
W20251024 19:48:25.840512 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d9367428428648f7ae07722ea09d70a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9367428428648f7ae07722ea09d70a7 already in progress: creating tablet
I20251024 19:48:25.844904 1369 raft_consensus.cc:3060] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:25.852334 1586 consensus_peers.cc:597] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:25.857214 1420 tablet.cc:2378] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 19:48:25.858927 1776 tablet_replica.cc:333] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:25.861933 1344 tablet_service.cc:1552] Processing DeleteTablet for tablet c084db163d874639b4385f41e9efaa18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.861907 1339 tablet_service.cc:1505] Processing CreateTablet for tablet b2a273b68eee4caeb468326187d6600c (DEFAULT_TABLE table=default.Table_50a7c4f2e9b94c2f9781f366cbac5b92 [id=fb47f2546fdc43e79bbbb9d581293c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.862323 1345 tablet_service.cc:1505] Processing CreateTablet for tablet ca38f3a765264dcc83f73a5d586cfea9 (DEFAULT_TABLE table=default.Table_92ca5409f01446e494834187689f234d [id=212636f6527a4647a129d04c6126cdc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.862802 1352 tablet_service.cc:1552] Processing DeleteTablet for tablet d72f4761820e430cb1a97d83cd30c75d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.862980 1343 tablet_service.cc:1552] Processing DeleteTablet for tablet b383a83425654723bd1d9c3a39b187be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.863065 1333 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c608ac55e9e42bc9c9b27beec717e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.863112 1341 tablet_service.cc:1552] Processing DeleteTablet for tablet e0228d5b788e4c42bf4ecce876ecc3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.863214 1348 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d26e02ad2945b39cf27738ed5b7cf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:25.863996 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 83bb9b2dbf5c48f38ed8b1f29d1ba466 (DEFAULT_TABLE table=default.Table_5046f8911611433295fd660877c0c7e3 [id=6d4415fcba5d41d5b77acd4c14e0ff78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.864426 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7e1ad08d4bd3477494cf5688f7842dd0" } new_table_name: "default.Table_319c59c379f548e0b483ab52e73664eb" modify_external_catalogs: true
I20251024 19:48:25.862993 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 5d62c5c336e14c65bf6734c66e9be198 (DEFAULT_TABLE table=default.Table_32612c9da3dd4bce8d35d2fdc3bb4bd4 [id=796ec6511bc64adeac0a19b4381b73ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.865756 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_311679c71ec1481e893a01440c1e07ca" } new_table_name: "default.Table_a4cfda713b2348a3ae3ae5ca8454d76b" modify_external_catalogs: true
I20251024 19:48:25.866842 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64499c8734d34c66b0b52a2cd3ef0673" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.866802 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f4b358360f27424397284c062bc6367e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.867306 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4b358360f27424397284c062bc6367e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.867704 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b22b58404b8b441ba97ef536885a66e0" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.868520 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d5bb186d0d74c8c8d135e04783d3a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.868508 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ce86da7ef95441c88843d4f307fdb809"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.868921 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce86da7ef95441c88843d4f307fdb809 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.862797 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 0f6f8cb6415d490184383c6f1ebe99f8 (DEFAULT_TABLE table=default.Table_1fcb39e601194b41bfaaa87ec866ae47 [id=7bdd81d3c7e64a1d91c64a156c9e63ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.873716 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f6f8cb6415d490184383c6f1ebe99f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.875660 1768 tablet_bootstrap.cc:654] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:25.830667 1473 tablet_service.cc:1505] Processing CreateTablet for tablet ca38f3a765264dcc83f73a5d586cfea9 (DEFAULT_TABLE table=default.Table_92ca5409f01446e494834187689f234d [id=212636f6527a4647a129d04c6126cdc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.830679 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 83bb9b2dbf5c48f38ed8b1f29d1ba466 (DEFAULT_TABLE table=default.Table_5046f8911611433295fd660877c0c7e3 [id=6d4415fcba5d41d5b77acd4c14e0ff78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.876623 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4c6bc454f1a4ffb975c3cd6ef9a491b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.876716 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca38f3a765264dcc83f73a5d586cfea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.876891 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e05eb2fb1ba742629c3e89f96630e986. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.831204 1486 tablet_service.cc:1552] Processing DeleteTablet for tablet c084db163d874639b4385f41e9efaa18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.831239 1478 tablet_service.cc:1505] Processing CreateTablet for tablet 5d62c5c336e14c65bf6734c66e9be198 (DEFAULT_TABLE table=default.Table_32612c9da3dd4bce8d35d2fdc3bb4bd4 [id=796ec6511bc64adeac0a19b4381b73ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.877521 1478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d62c5c336e14c65bf6734c66e9be198. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.831331 1481 tablet_service.cc:1552] Processing DeleteTablet for tablet d72f4761820e430cb1a97d83cd30c75d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.831368 1484 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c608ac55e9e42bc9c9b27beec717e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.881703 1483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83bb9b2dbf5c48f38ed8b1f29d1ba466. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.883890 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca38f3a765264dcc83f73a5d586cfea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.889496 1774 tablet_replica.cc:333] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:25.890244 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a273b68eee4caeb468326187d6600c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.890426 1775 ts_tablet_manager.cc:1950] T 0266cef4e2034303978d7456c47ea98c P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:25.890765 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1026) took 5476 ms. Trace:
I20251024 19:48:25.890834 1775 rpcz_store.cc:276] 1024 19:48:20.414400 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.414492 (+ 92us) service_pool.cc:225] Handling call
1024 19:48:25.701981 (+5287489us) ts_tablet_manager.cc:1151] Deleting tablet 0266cef4e2034303978d7456c47ea98c
1024 19:48:25.701991 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.767883 (+ 65892us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:25.829158 (+ 61275us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:25.890742 (+ 61584us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5274618}
I20251024 19:48:25.891247 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1027) took 5467 ms. Trace:
I20251024 19:48:25.891316 1775 rpcz_store.cc:276] 1024 19:48:20.423313 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.423373 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:25.891114 (+5467741us) ts_tablet_manager.cc:1151] Deleting tablet fc187c8f34594f4fb37d8afe0855263b
1024 19:48:25.891131 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.891232 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5463698}
I20251024 19:48:25.891546 1775 tablet_replica.cc:333] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
W20251024 19:48:25.891577 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d391bc3b3a144415bd4e6b5fdcc37612 consensus state has no leader
I20251024 19:48:25.891712 1775 raft_consensus.cc:2243] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:25.891884 1775 raft_consensus.cc:2272] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:25.891947 1329 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:54172 (request call id 171) took 1029 ms. Trace:
I20251024 19:48:25.892019 1329 rpcz_store.cc:276] 1024 19:48:24.862190 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.862245 (+ 55us) service_pool.cc:225] Handling call
1024 19:48:25.039354 (+177109us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x616000154654. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:25.469015 (+429661us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x616000112654. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:25.891924 (+422909us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":350840064}
I20251024 19:48:25.892418 1775 ts_tablet_manager.cc:1916] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:25.893548 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83bb9b2dbf5c48f38ed8b1f29d1ba466. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.893749 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c910a21a05ea4d3a8d8a400a3cc58f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.897761 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d62c5c336e14c65bf6734c66e9be198. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:25.898756 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 830528cabe344ce89e5092dae613c7fe has no consensus state
I20251024 19:48:25.900125 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 2e7b3288182744969e217d4f174fe4ea (table default.Table_c4882268578941ccaf2c28be3074fdd2 [id=0b00aa7d2eaa4b1a967952b9a484fd42]) successfully deleted
W20251024 19:48:25.900388 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 7e747880c4e1425ba749201e624a083e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e747880c4e1425ba749201e624a083e already in progress: creating tablet
I20251024 19:48:25.902034 3456 raft_consensus.cc:493] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.902211 3456 raft_consensus.cc:515] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.902997 3456 leader_election.cc:290] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:25.903664 1586 consensus_peers.cc:597] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:25.925549 3455 raft_consensus.cc:493] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.925729 3455 raft_consensus.cc:515] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.926328 3455 leader_election.cc:290] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.926437 3455 raft_consensus.cc:493] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.926524 3455 raft_consensus.cc:515] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:25.927081 3455 leader_election.cc:290] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:25.928928 1642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1d26e02ad2945b39cf27738ed5b7cf4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:25.929706 1305 leader_election.cc:304] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.929982 3457 raft_consensus.cc:2749] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:25.930757 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 0266cef4e2034303978d7456c47ea98c (table default.Table_74e6a6d8b7a34049994d41a379a9a9f0 [id=7f87fee0587e4cb5a4f6f5afb3238334]) successfully deleted
W20251024 19:48:25.931198 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet fc187c8f34594f4fb37d8afe0855263b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc187c8f34594f4fb37d8afe0855263b already in progress: creating tablet
I20251024 19:48:25.931953 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30bce965315d43948ecb8282c829c557" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.935262 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccd46da86fe444baa675d3391cdd410c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:25.936158 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73278db2097d4001a538d62dacb97c9b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:25.936725 1447 consensus_peers.cc:597] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e7b3288182744969e217d4f174fe4ea. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:25.936928 1447 leader_election.cc:343] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: a3259823b622461fa823841925b3d203
W20251024 19:48:25.937202 1447 leader_election.cc:343] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.937299 1447 leader_election.cc:304] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.834630 1490 tablet_service.cc:1505] Processing CreateTablet for tablet b2a273b68eee4caeb468326187d6600c (DEFAULT_TABLE table=default.Table_50a7c4f2e9b94c2f9781f366cbac5b92 [id=fb47f2546fdc43e79bbbb9d581293c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.938005 3456 raft_consensus.cc:2749] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:25.938140 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a273b68eee4caeb468326187d6600c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.846194 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 398e73d6d576487e968c9083175965aa (DEFAULT_TABLE table=default.Table_40837137d2a145828e6dcbd42e88e393 [id=a174648aa0f944d193fde7710520a5e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:25.846808 1480 tablet_service.cc:1552] Processing DeleteTablet for tablet b383a83425654723bd1d9c3a39b187be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.847414 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet e0228d5b788e4c42bf4ecce876ecc3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:25.945015 3510 tablet.cc:1722] T f965111ac60841dbb9e6ecc21ff3fec0 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:25.945973 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2549427b0e704654a43073c7ef97be52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:25.946331 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_03461325877345a4aaa7e0437abd8813"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.946568 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2549427b0e704654a43073c7ef97be52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:25.946707 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03461325877345a4aaa7e0437abd8813 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.946007 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4ae2544614374b45bd7da7facda94f2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:25.947849 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7ab1e070bda4405686f54e794245f399"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.947942 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ae2544614374b45bd7da7facda94f2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:25.948272 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ab1e070bda4405686f54e794245f399 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.949316 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_12056492244041dabd5c64f523c27d7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.949777 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12056492244041dabd5c64f523c27d7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.951102 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0b4fadd9a67b4f43965ba55cba4fe64e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.951491 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b4fadd9a67b4f43965ba55cba4fe64e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.956395 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 1999b751e9344dcf81d26226c0589c8f (table default.Table_9147a98b3bf9483ea7b7a6f17130d4fc [id=d743705c21da4b89abedf2eb34ddbe09]). Expected version 1 got 0
I20251024 19:48:25.956797 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (table default.Table_18e7285b7df44399ab3be1dfc86eb91f [id=14bda7f9324f44aa9cb87a5f129039be]). Expected version 2 got 0
I20251024 19:48:25.954404 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 398e73d6d576487e968c9083175965aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:25.958256 1776 raft_consensus.cc:2243] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:25.958695 1776 pending_rounds.cc:70] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:25.958801 1776 pending_rounds.cc:77] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429399049613312 op_type: NO_OP noop_request { }
I20251024 19:48:25.958910 3456 raft_consensus.cc:493] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:25.958985 1776 raft_consensus.cc:2889] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:25.959067 1776 raft_consensus.cc:2272] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:25.959026 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d1092b6bac8f4c90a5eb12dd16060570" } new_table_name: "default.Table_002b63cb91e34275b89bb2f67c5e5969" modify_external_catalogs: true
I20251024 19:48:25.959036 3456 raft_consensus.cc:515] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:25.959800 1776 ts_tablet_manager.cc:1916] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:25.959821 3456 leader_election.cc:290] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:25.961980 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5b894129f4b642909eeb0fe89ec44dea" } modify_external_catalogs: true
I20251024 19:48:25.962205 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5b894129f4b642909eeb0fe89ec44dea" } modify_external_catalogs: true
I20251024 19:48:25.963387 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_bc3b64103c5b40db85bb106ea58c5c62" } new_table_name: "default.Table_f6f3683609784447acb05ef28817aa01" modify_external_catalogs: true
I20251024 19:48:25.970757 1766 tablet_bootstrap.cc:492] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:25.970924 1766 ts_tablet_manager.cc:1403] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.214s user 0.003s sys 0.000s
I20251024 19:48:25.971628 1766 raft_consensus.cc:359] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.971817 1766 raft_consensus.cc:385] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:25.971873 1766 raft_consensus.cc:740] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:25.972102 1766 consensus_queue.cc:260] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:25.972685 1766 ts_tablet_manager.cc:1434] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:25.972983 1766 tablet_bootstrap.cc:492] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:25.973397 1768 tablet_bootstrap.cc:492] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:25.973551 1768 ts_tablet_manager.cc:1403] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.355s user 0.001s sys 0.004s
I20251024 19:48:25.974380 1768 raft_consensus.cc:359] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:25.974542 1768 raft_consensus.cc:385] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:25.974604 1768 raft_consensus.cc:740] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:25.974828 1768 consensus_queue.cc:260] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:25.975320 1768 ts_tablet_manager.cc:1434] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:25.975605 1768 tablet_bootstrap.cc:492] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:25.976188 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d6fedd4e6fbf4a718b3b5fd14d74f48b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:25.976706 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6fedd4e6fbf4a718b3b5fd14d74f48b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:25.977007 1768 tablet_bootstrap.cc:654] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:25.978844 1447 leader_election.cc:343] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.978957 1447 leader_election.cc:304] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:25.979189 3456 raft_consensus.cc:2749] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:25.979322 1447 leader_election.cc:343] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:25.985848 1447 leader_election.cc:343] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:25.986429 1447 leader_election.cc:343] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:25.986939 3509 tablet.cc:1722] T f965111ac60841dbb9e6ecc21ff3fec0 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:26.001860 3524 tablet.cc:1722] T f965111ac60841dbb9e6ecc21ff3fec0 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:26.006882 1776 ts_tablet_manager.cc:1929] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.007108 1774 raft_consensus.cc:2243] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.007797 1776 log.cc:1199] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/d2a68bb1dc6c483388784b9bb19ce065
I20251024 19:48:26.008322 1776 ts_tablet_manager.cc:1950] T d2a68bb1dc6c483388784b9bb19ce065 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:26.011803 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1080) took 3288 ms. Trace:
I20251024 19:48:26.011979 1776 rpcz_store.cc:276] 1024 19:48:22.723529 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.723590 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:25.858849 (+3135259us) ts_tablet_manager.cc:1151] Deleting tablet d2a68bb1dc6c483388784b9bb19ce065
1024 19:48:25.858866 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:25.991606 (+132740us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.006701 (+ 15095us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.011777 (+ 5076us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3099711}
I20251024 19:48:26.012588 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet d2a68bb1dc6c483388784b9bb19ce065 (table default.Table_3be1c2c5954947969e6903a3f016d589 [id=0b304e4408b24fe797f4438b874e480f]) successfully deleted
I20251024 19:48:26.018496 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1081) took 3280 ms. Trace:
I20251024 19:48:26.018657 1776 rpcz_store.cc:276] 1024 19:48:22.738235 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.738583 (+ 348us) service_pool.cc:225] Handling call
1024 19:48:26.012343 (+3273760us) ts_tablet_manager.cc:1151] Deleting tablet 4e14a82420f24157a36df44d050b88bf
1024 19:48:26.018297 (+ 5954us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.018463 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3248217}
I20251024 19:48:26.019161 1774 raft_consensus.cc:2272] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:26.019493 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4e14a82420f24157a36df44d050b88bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e14a82420f24157a36df44d050b88bf already in progress: creating tablet
I20251024 19:48:26.022696 1774 ts_tablet_manager.cc:1916] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:26.027407 1586 consensus_peers.cc:597] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38818c7997c84cedad5f9b09eef77105. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:26.028363 1767 tablet_bootstrap.cc:492] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:26.028546 1767 ts_tablet_manager.cc:1403] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.429s user 0.006s sys 0.010s
I20251024 19:48:26.029273 1767 raft_consensus.cc:359] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.029438 1767 raft_consensus.cc:385] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.029556 1767 raft_consensus.cc:740] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.029851 1767 consensus_queue.cc:260] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.030356 1767 ts_tablet_manager.cc:1434] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251024 19:48:25.989682 3456 raft_consensus.cc:493] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:26.015349 1766 tablet_bootstrap.cc:654] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.035562 3456 raft_consensus.cc:515] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:26.036302 3456 leader_election.cc:290] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:26.037084 1447 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 50661.5us
I20251024 19:48:26.038530 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6f8764e9ae472b9ac35a032182aed1" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:26.042980 1774 ts_tablet_manager.cc:1929] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.043116 1774 log.cc:1199] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/0266cef4e2034303978d7456c47ea98c
I20251024 19:48:26.043191 1645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6f8764e9ae472b9ac35a032182aed1" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:26.043304 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccd46da86fe444baa675d3391cdd410c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:26.043152 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30bce965315d43948ecb8282c829c557" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:26.043435 1644 raft_consensus.cc:2468] T ccd46da86fe444baa675d3391cdd410c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:26.043459 1646 raft_consensus.cc:2468] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:26.043511 1774 ts_tablet_manager.cc:1950] T 0266cef4e2034303978d7456c47ea98c P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:26.044052 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebcc09d8e029451d8fb6b6601ad16087" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:26.044649 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73278db2097d4001a538d62dacb97c9b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:26.045250 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1082) took 3237 ms. Trace:
I20251024 19:48:26.047483 1774 tablet_replica.cc:333] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:26.047690 1774 raft_consensus.cc:2243] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.047834 1774 raft_consensus.cc:2272] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.048343 1774 ts_tablet_manager.cc:1916] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.054555 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 0266cef4e2034303978d7456c47ea98c (table default.Table_74e6a6d8b7a34049994d41a379a9a9f0 [id=7f87fee0587e4cb5a4f6f5afb3238334]) successfully deleted
W20251024 19:48:26.054930 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4100abc9ff12471e9a4959b6b24f7e99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4100abc9ff12471e9a4959b6b24f7e99 already in progress: creating tablet
W20251024 19:48:26.055222 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 28f93be6762146398421ac19d2e84e0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28f93be6762146398421ac19d2e84e0c already in progress: creating tablet
W20251024 19:48:26.055459 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 5ace18d554f14acc892bc956c30e9979 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ace18d554f14acc892bc956c30e9979 already in progress: creating tablet
W20251024 19:48:26.055671 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 977b4070419948d4910e2b69e01ff132 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 977b4070419948d4910e2b69e01ff132 already in progress: creating tablet
W20251024 19:48:26.055939 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet bec3806757214c869cf1c4cecd7cc26e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bec3806757214c869cf1c4cecd7cc26e already in progress: creating tablet
W20251024 19:48:26.058350 1444 leader_election.cc:343] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: a3259823b622461fa823841925b3d203
I20251024 19:48:26.058485 1444 leader_election.cc:304] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:26.058674 1444 raft_consensus.cc:2686] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251024 19:48:26.058689 1774 ts_tablet_manager.cc:1929] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:26.058813 1774 log.cc:1199] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/45cff9bcc4924e0081518c284f074940
I20251024 19:48:26.059087 1774 ts_tablet_manager.cc:1950] T 45cff9bcc4924e0081518c284f074940 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:48:26.059744 1444 leader_election.cc:343] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.060431 3530 raft_consensus.cc:493] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:26.060436 1444 leader_election.cc:304] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:26.060544 1774 tablet_replica.cc:333] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:26.060570 3530 raft_consensus.cc:515] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.060703 1774 raft_consensus.cc:2243] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.060838 1774 raft_consensus.cc:2272] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.061132 3456 raft_consensus.cc:2804] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:26.061220 3456 raft_consensus.cc:493] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:26.061415 3456 raft_consensus.cc:3060] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.061546 3530 leader_election.cc:290] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:26.063439 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b632acc1348e49f8a1eea019c007780c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:26.063668 1511 raft_consensus.cc:2468] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:26.064356 1308 leader_election.cc:304] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:26.064646 3530 raft_consensus.cc:2804] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:26.064762 3530 raft_consensus.cc:493] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:26.064844 3530 raft_consensus.cc:3060] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.065469 3456 raft_consensus.cc:515] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.066176 3456 leader_election.cc:290] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:26.067174 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 45cff9bcc4924e0081518c284f074940 (table default.Table_34ef66bee6bd460b92cf30b469b3f179 [id=32d4cdae3aac408a88b0a207c8c08fcd]) successfully deleted
W20251024 19:48:26.067428 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet c084db163d874639b4385f41e9efaa18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c084db163d874639b4385f41e9efaa18 already in progress: creating tablet
W20251024 19:48:26.067601 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet d72f4761820e430cb1a97d83cd30c75d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d72f4761820e430cb1a97d83cd30c75d already in progress: creating tablet
W20251024 19:48:26.067768 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b383a83425654723bd1d9c3a39b187be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b383a83425654723bd1d9c3a39b187be already in progress: creating tablet
W20251024 19:48:26.067936 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3c608ac55e9e42bc9c9b27beec717e30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c608ac55e9e42bc9c9b27beec717e30 already in progress: creating tablet
I20251024 19:48:26.069152 3530 raft_consensus.cc:515] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.070076 1444 leader_election.cc:304] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
W20251024 19:48:26.070788 1444 leader_election.cc:343] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:26.070984 1444 leader_election.cc:343] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.071074 1444 leader_election.cc:304] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:26.071461 2858 raft_consensus.cc:2804] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:26.071568 2858 raft_consensus.cc:493] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:26.071650 2858 raft_consensus.cc:3060] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.072786 3390 raft_consensus.cc:2749] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:26.073242 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebcc09d8e029451d8fb6b6601ad16087" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:26.073995 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30bce965315d43948ecb8282c829c557" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:26.074030 1447 leader_election.cc:343] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.074112 1774 ts_tablet_manager.cc:1916] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.074200 1447 leader_election.cc:304] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:26.074669 3390 raft_consensus.cc:2749] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:26.075141 1447 consensus_peers.cc:597] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81f8393f78204860bd854f4872099387. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:26.075383 1447 leader_election.cc:343] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.075541 1447 leader_election.cc:304] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:26.075788 3390 raft_consensus.cc:2749] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:26.076234 1447 leader_election.cc:343] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.049779 3528 raft_consensus.cc:493] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:26.076871 3528 raft_consensus.cc:515] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.064249 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b632acc1348e49f8a1eea019c007780c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:26.077704 1643 raft_consensus.cc:2468] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:26.078154 3528 leader_election.cc:290] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:26.078621 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b632acc1348e49f8a1eea019c007780c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:26.078706 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b632acc1348e49f8a1eea019c007780c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:26.078989 1511 raft_consensus.cc:2468] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:26.078882 1369 raft_consensus.cc:2393] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 43bec2f03fc849099e254f6335e9d532 in this term.
I20251024 19:48:26.079424 1586 leader_election.cc:304] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:26.079463 3530 leader_election.cc:290] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:26.079691 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b632acc1348e49f8a1eea019c007780c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:26.079867 1643 raft_consensus.cc:3060] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.083384 1643 raft_consensus.cc:2468] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:26.084168 3528 raft_consensus.cc:2764] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251024 19:48:26.084556 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30bce965315d43948ecb8282c829c557" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:26.084827 1647 raft_consensus.cc:3060] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.085139 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b632acc1348e49f8a1eea019c007780c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:26.085330 1511 raft_consensus.cc:3060] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.085747 1774 ts_tablet_manager.cc:1929] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:26.085858 1774 log.cc:1199] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/d1d26e02ad2945b39cf27738ed5b7cf4
I20251024 19:48:26.086163 1774 ts_tablet_manager.cc:1950] T d1d26e02ad2945b39cf27738ed5b7cf4 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:26.086315 1776 rpcz_store.cc:276] 1024 19:48:22.807502 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.807562 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:26.019505 (+3211943us) ts_tablet_manager.cc:1151] Deleting tablet 4e14a82420f24157a36df44d050b88bf
1024 19:48:26.042789 (+ 23284us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.045229 (+ 2440us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3211701}
I20251024 19:48:26.086771 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1098) took 2988 ms. Trace:
I20251024 19:48:26.086844 1776 rpcz_store.cc:276] 1024 19:48:23.098254 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.098302 (+ 48us) service_pool.cc:225] Handling call
1024 19:48:26.086628 (+2988326us) ts_tablet_manager.cc:1151] Deleting tablet 80688d7b8a394a2a910c4b8910fe4cf2
1024 19:48:26.086640 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.086751 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2927033}
I20251024 19:48:26.087142 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1099) took 2988 ms. Trace:
I20251024 19:48:26.087204 1776 rpcz_store.cc:276] 1024 19:48:23.098637 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.098695 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:26.087041 (+2988346us) ts_tablet_manager.cc:1151] Deleting tablet 83d34ec47c2845058e870c72cc832e9e
1024 19:48:26.087050 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.087130 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2927380}
I20251024 19:48:26.087462 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1100) took 2988 ms. Trace:
I20251024 19:48:26.087539 1776 rpcz_store.cc:276] 1024 19:48:23.098911 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.099252 (+ 341us) service_pool.cc:225] Handling call
1024 19:48:26.087378 (+2988126us) ts_tablet_manager.cc:1151] Deleting tablet 9198321bcc9e4e47a4c2a5442b6bc850
1024 19:48:26.087386 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.087451 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2927664}
I20251024 19:48:26.087816 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1101) took 2988 ms. Trace:
I20251024 19:48:26.087874 1776 rpcz_store.cc:276] 1024 19:48:23.098986 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.099405 (+ 419us) service_pool.cc:225] Handling call
1024 19:48:26.087711 (+2988306us) ts_tablet_manager.cc:1151] Deleting tablet 9b7ab1bf5b9549afb150802a209527eb
1024 19:48:26.087718 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.087803 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2927945}
I20251024 19:48:26.088079 1776 tablet_replica.cc:333] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:26.088243 1776 raft_consensus.cc:2243] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:26.088587 1776 raft_consensus.cc:2272] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.088809 1647 raft_consensus.cc:2468] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:26.089143 1776 ts_tablet_manager.cc:1916] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:26.090137 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4e14a82420f24157a36df44d050b88bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e14a82420f24157a36df44d050b88bf already in progress: creating tablet
I20251024 19:48:26.090301 1305 leader_election.cc:304] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:26.090390 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 80688d7b8a394a2a910c4b8910fe4cf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80688d7b8a394a2a910c4b8910fe4cf2 already in progress: creating tablet
W20251024 19:48:26.090580 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 83d34ec47c2845058e870c72cc832e9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83d34ec47c2845058e870c72cc832e9e already in progress: creating tablet
W20251024 19:48:26.090749 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9198321bcc9e4e47a4c2a5442b6bc850 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9198321bcc9e4e47a4c2a5442b6bc850 already in progress: creating tablet
W20251024 19:48:26.090955 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9b7ab1bf5b9549afb150802a209527eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b7ab1bf5b9549afb150802a209527eb already in progress: creating tablet
I20251024 19:48:26.090924 1444 leader_election.cc:304] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:26.091805 3456 raft_consensus.cc:2804] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:26.091964 3456 raft_consensus.cc:697] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:26.092459 3530 raft_consensus.cc:2804] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:26.092604 3530 raft_consensus.cc:697] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
W20251024 19:48:26.092963 1177 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.254:36505, user_credentials={real_user=slave}} blocked reactor thread for 105588us
I20251024 19:48:26.093032 3530 consensus_queue.cc:237] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.108918 2858 raft_consensus.cc:515] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.109850 2858 leader_election.cc:290] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:26.112888 1190 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet f965111ac60841dbb9e6ecc21ff3fec0 (table default.Table_b5386eb425a14340919f7bbae554508f [id=c727179cd01648cd86e1d9eca2396cae]). Expected version 1 got 0
I20251024 19:48:26.113044 1511 raft_consensus.cc:2468] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:26.092276 3456 consensus_queue.cc:237] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.116171 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccd46da86fe444baa675d3391cdd410c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:26.116446 1647 raft_consensus.cc:3060] T ccd46da86fe444baa675d3391cdd410c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.113327 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_94fcb02d0c374222a5ceb6163a6b9f2f" } modify_external_catalogs: true
I20251024 19:48:26.123157 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_94fcb02d0c374222a5ceb6163a6b9f2f" } modify_external_catalogs: true
I20251024 19:48:26.113821 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_56c54a42391a4397bdbcc0964ed0f9e7" } modify_external_catalogs: true
I20251024 19:48:26.123688 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_56c54a42391a4397bdbcc0964ed0f9e7" } modify_external_catalogs: true
I20251024 19:48:26.114114 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_32612c9da3dd4bce8d35d2fdc3bb4bd4" } modify_external_catalogs: true
I20251024 19:48:26.124961 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_32612c9da3dd4bce8d35d2fdc3bb4bd4" } modify_external_catalogs: true
I20251024 19:48:26.126363 1190 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 1999b751e9344dcf81d26226c0589c8f (table default.Table_9147a98b3bf9483ea7b7a6f17130d4fc [id=d743705c21da4b89abedf2eb34ddbe09]). Expected version 1 got 0
I20251024 19:48:26.121456 1647 raft_consensus.cc:2468] T ccd46da86fe444baa675d3391cdd410c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:26.118786 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f57b8c8910c2489a80979e2ffe453f3c" } modify_external_catalogs: true
W20251024 19:48:26.120769 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71ab984ce3124f58a059c5fd4dc29b55 consensus state has no leader
I20251024 19:48:26.129187 1444 leader_election.cc:304] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:26.129784 2858 raft_consensus.cc:2804] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:26.130003 2858 raft_consensus.cc:697] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:26.130398 2858 consensus_queue.cc:237] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.130990 1189 catalog_manager.cc:5649] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43bec2f03fc849099e254f6335e9d532 (127.0.252.193). New cstate: current_term: 1 leader_uuid: "43bec2f03fc849099e254f6335e9d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: HEALTHY } } }
I20251024 19:48:26.131388 1189 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet b632acc1348e49f8a1eea019c007780c (table default.Table_361840e201d943c18014ec71a255cc9c [id=dc00ee30038e49fb9f86499cfb985a0e]). Expected version 2 got 0
I20251024 19:48:26.131611 1189 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet a1fbbba88a294adfbf97c0343d86698e (table default.Table_7bc8cd4a930b44dfafa473297875594d [id=a60c8222aa0248c091fab8dc34100847]). Expected version 2 got 0
I20251024 19:48:26.132484 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f57b8c8910c2489a80979e2ffe453f3c" } modify_external_catalogs: true
I20251024 19:48:26.137992 3390 consensus_queue.cc:1048] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:26.139835 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 0266cef4e2034303978d7456c47ea98c with delete_type TABLET_DATA_DELETED (replaced by tablet 64325d89db714554866e74922d103e34) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:26.142586 1369 raft_consensus.cc:3060] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.146129 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet d1d26e02ad2945b39cf27738ed5b7cf4 (table default.Table_2c02fd9cab9b479d8876a6dcbd586b17 [id=3660d29be985453fa1741ee293ac2a14]) successfully deleted
I20251024 19:48:26.146335 3528 consensus_queue.cc:1048] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:26.146692 1767 tablet_bootstrap.cc:492] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:26.148290 1767 tablet_bootstrap.cc:654] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:26.151373 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 169227us
W20251024 19:48:26.154780 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet e0228d5b788e4c42bf4ecce876ecc3f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0228d5b788e4c42bf4ecce876ecc3f4 already in progress: creating tablet
W20251024 19:48:26.157356 1179 catalog_manager.cc:4952] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 0266cef4e2034303978d7456c47ea98c because the tablet was not found. No further retry: Not found: Tablet not found: 0266cef4e2034303978d7456c47ea98c
W20251024 19:48:26.158879 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 505ba3165385434faea556e97f42a7f6 consensus state has no leader
I20251024 19:48:26.163007 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_12d28513e70345139f2f009dc4259026"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:26.163705 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12d28513e70345139f2f009dc4259026 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:26.162202 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_423e7aaec4b54cb79a21504a67d8b143" } new_table_name: "default.Table_08713f9a63294379bf4f2e7f2f635906" modify_external_catalogs: true
I20251024 19:48:26.166349 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccd46da86fe444baa675d3391cdd410c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:26.187723 1447 leader_election.cc:343] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:26.191036 1447 consensus_peers.cc:597] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:26.194835 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet bcdd47a64e2142cda1f2d20afebd4ce5
I20251024 19:48:26.196434 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet bcdd47a64e2142cda1f2d20afebd4ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:26.203709 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3cd13b4df8f54c22aad4df0a6ccaef3e" } new_table_name: "default.Table_1260f1d3582f4590983f08c4a7d0535f" modify_external_catalogs: true
I20251024 19:48:26.209403 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet bcdd47a64e2142cda1f2d20afebd4ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:26.211375 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 5d62c5c336e14c65bf6734c66e9be198
I20251024 19:48:26.213120 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d62c5c336e14c65bf6734c66e9be198 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:26.216325 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d62c5c336e14c65bf6734c66e9be198 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:26.216781 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet bcdd47a64e2142cda1f2d20afebd4ce5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcdd47a64e2142cda1f2d20afebd4ce5 already in progress: creating tablet
W20251024 19:48:26.217068 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 5d62c5c336e14c65bf6734c66e9be198 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d62c5c336e14c65bf6734c66e9be198 already in progress: creating tablet
I20251024 19:48:26.218837 1768 tablet_bootstrap.cc:492] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:26.219179 1768 ts_tablet_manager.cc:1403] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.244s user 0.002s sys 0.002s
I20251024 19:48:26.220235 1768 raft_consensus.cc:359] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.222011 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 5d62c5c336e14c65bf6734c66e9be198 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:26.222656 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f4b358360f27424397284c062bc6367e" } new_table_name: "default.Table_b32e44b4fac24495b6c62f393550ec11" modify_external_catalogs: true
I20251024 19:48:26.223222 1768 raft_consensus.cc:385] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.222011 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet bcdd47a64e2142cda1f2d20afebd4ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:26.223589 1768 raft_consensus.cc:740] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.224205 1768 consensus_queue.cc:260] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.225790 1768 ts_tablet_manager.cc:1434] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
W20251024 19:48:26.229959 1698 tablet.cc:2378] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 19:48:26.231292 1775 ts_tablet_manager.cc:1929] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 19:48:26.231582 1775 log.cc:1199] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/a3259823b622461fa823841925b3d203
I20251024 19:48:26.232139 1775 ts_tablet_manager.cc:1950] T a3259823b622461fa823841925b3d203 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:26.233278 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1028) took 5742 ms. Trace:
I20251024 19:48:26.241653 1187 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_b5386eb425a14340919f7bbae554508f [id=c727179cd01648cd86e1d9eca2396cae] alter complete (version 1)
I20251024 19:48:26.244042 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet d2a68bb1dc6c483388784b9bb19ce065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:26.249009 1768 tablet_bootstrap.cc:492] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:26.251889 1768 tablet_bootstrap.cc:654] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.254427 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c18a4e82a4db4df5a237651828a095c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:26.255162 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c18a4e82a4db4df5a237651828a095c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:26.267565 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b with delete_type TABLET_DATA_DELETED (replaced by tablet 6bebcdb80476423b8e6c750f749a1771) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:26.267910 1774 tablet_replica.cc:333] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:26.268111 1774 raft_consensus.cc:2243] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.268272 1774 pending_rounds.cc:70] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:26.268307 1774 pending_rounds.cc:77] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429410451877888 op_type: NO_OP noop_request { }
I20251024 19:48:26.268409 1774 raft_consensus.cc:2889] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:26.268493 1774 raft_consensus.cc:2272] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.273988 1775 rpcz_store.cc:276] 1024 19:48:20.490843 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.490922 (+ 79us) service_pool.cc:225] Handling call
1024 19:48:25.891507 (+5400585us) ts_tablet_manager.cc:1151] Deleting tablet a3259823b622461fa823841925b3d203
1024 19:48:25.891515 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.058711 (+167196us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.231120 (+172409us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.233248 (+ 2128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5333304}
I20251024 19:48:26.274264 1774 ts_tablet_manager.cc:1916] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.277261 1189 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 10891ff5bf0d4b93bcab9faa5f7346f1 (table default.Table_18e7285b7df44399ab3be1dfc86eb91f [id=14bda7f9324f44aa9cb87a5f129039be]). Expected version 2 got 0
I20251024 19:48:26.277544 1189 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 1999b751e9344dcf81d26226c0589c8f (table default.Table_9147a98b3bf9483ea7b7a6f17130d4fc [id=d743705c21da4b89abedf2eb34ddbe09]). Expected version 1 got 0
I20251024 19:48:26.277812 1189 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 2d01572e51694d9bb58c3a59d4205102 (table default.Table_bf15d3c700774490b1860d3d8865a95e [id=2ea13e78137d47429d7cbe374d2113ef]). Expected version 1 got 0
I20251024 19:48:26.281868 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 30bce965315d43948ecb8282c829c557 with delete_type TABLET_DATA_DELETED (replaced by tablet 043fac06e6324032a61c42a10d51ea22) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:26.282752 1774 ts_tablet_manager.cc:1929] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.282918 1774 log.cc:1199] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/3fd8aa1d9aca429dbfe46f472ebbf53b
I20251024 19:48:26.283699 1774 ts_tablet_manager.cc:1950] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
W20251024 19:48:26.286096 1586 consensus_peers.cc:597] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:26.289611 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 3fd8aa1d9aca429dbfe46f472ebbf53b (table default.Table_003cb0c156394f9b8b0ba1b4ac52ce4c [id=b81a7d71911844c1be495ade2ceaf1d4]) successfully deleted
I20251024 19:48:26.289883 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet a3259823b622461fa823841925b3d203 (table default.Table_ea0d3589c13a4ebeb950ddc737a9948b [id=350b37bf42194fcfbb660bffe83e47c4]) successfully deleted
I20251024 19:48:26.298013 3360 consensus_queue.cc:1048] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:26.298430 1775 tablet_replica.cc:333] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:26.298655 1775 raft_consensus.cc:2243] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.298821 1775 raft_consensus.cc:2272] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.298998 1643 raft_consensus.cc:1275] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Refusing update from remote peer 43bec2f03fc849099e254f6335e9d532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 19:48:26.299134 1509 raft_consensus.cc:1275] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Refusing update from remote peer 43bec2f03fc849099e254f6335e9d532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 19:48:26.299883 3530 consensus_queue.cc:1048] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:26.301671 3530 consensus_queue.cc:1048] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:26.308444 1767 tablet_bootstrap.cc:492] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:26.308634 1767 ts_tablet_manager.cc:1403] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.162s user 0.003s sys 0.000s
I20251024 19:48:26.309456 1767 raft_consensus.cc:359] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.309836 1767 raft_consensus.cc:385] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.309900 1767 raft_consensus.cc:740] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.310103 1767 consensus_queue.cc:260] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.311188 1768 tablet_bootstrap.cc:492] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:26.311328 1768 ts_tablet_manager.cc:1403] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.001s
I20251024 19:48:26.312011 1768 raft_consensus.cc:359] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.312150 1767 ts_tablet_manager.cc:1434] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:26.312175 1768 raft_consensus.cc:385] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.312222 1768 raft_consensus.cc:740] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.312435 1767 tablet_bootstrap.cc:492] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:26.312394 1768 consensus_queue.cc:260] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.312813 1768 ts_tablet_manager.cc:1434] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251024 19:48:26.313858 1767 tablet_bootstrap.cc:654] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.307649 1775 ts_tablet_manager.cc:1916] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.314898 1768 tablet_bootstrap.cc:492] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:26.318162 3524 tablet.cc:1722] T b632acc1348e49f8a1eea019c007780c P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:26.319284 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet c5cbce9e4b61451ba7f458df922be759
I20251024 19:48:26.319583 1768 tablet_bootstrap.cc:654] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.322310 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet c5cbce9e4b61451ba7f458df922be759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:26.324819 1586 consensus_peers.cc:597] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:26.329102 3509 tablet.cc:1722] T b632acc1348e49f8a1eea019c007780c P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:26.334262 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet c5cbce9e4b61451ba7f458df922be759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:26.334993 1774 tablet_replica.cc:333] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:26.335186 1774 raft_consensus.cc:2243] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.335369 1774 raft_consensus.cc:2272] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.335575 3538 raft_consensus.cc:493] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:26.335711 3538 raft_consensus.cc:515] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:26.336059 1774 ts_tablet_manager.cc:1916] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.336156 1768 tablet_bootstrap.cc:492] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:26.336272 1768 ts_tablet_manager.cc:1403] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20251024 19:48:26.336361 3538 leader_election.cc:290] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:26.336954 1768 raft_consensus.cc:359] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.337106 1768 raft_consensus.cc:385] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.337162 1768 raft_consensus.cc:740] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.337237 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92d78642772d44179e67d75c51b9da4e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:26.337771 1447 leader_election.cc:343] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.341917 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b693a6903f7e41da99547f6fb43144c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:26.342618 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b693a6903f7e41da99547f6fb43144c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:26.343322 1774 ts_tablet_manager.cc:1929] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.343478 1774 log.cc:1199] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/c5cbce9e4b61451ba7f458df922be759
I20251024 19:48:26.343909 1774 ts_tablet_manager.cc:1950] T c5cbce9e4b61451ba7f458df922be759 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:26.346016 1768 consensus_queue.cc:260] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.348666 1776 ts_tablet_manager.cc:1929] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.346566 1768 ts_tablet_manager.cc:1434] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251024 19:48:26.349815 1768 tablet_bootstrap.cc:492] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:26.359373 1768 tablet_bootstrap.cc:654] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.361783 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet c5cbce9e4b61451ba7f458df922be759 (table default.Table_f57b8c8910c2489a80979e2ffe453f3c [id=1817b735a20f46409680f85379280ca6]) successfully deleted
I20251024 19:48:26.362520 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b with delete_type TABLET_DATA_DELETED (replaced by tablet 6bebcdb80476423b8e6c750f749a1771) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:26.364282 1776 log.cc:1199] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/0266cef4e2034303978d7456c47ea98c
I20251024 19:48:26.364711 1776 ts_tablet_manager.cc:1950] T 0266cef4e2034303978d7456c47ea98c P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
W20251024 19:48:26.365172 1179 catalog_manager.cc:4952] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3fd8aa1d9aca429dbfe46f472ebbf53b because the tablet was not found. No further retry: Not found: Tablet not found: 3fd8aa1d9aca429dbfe46f472ebbf53b
I20251024 19:48:26.365594 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1103) took 3266 ms. Trace:
I20251024 19:48:26.365671 1776 rpcz_store.cc:276] 1024 19:48:23.099558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.099604 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:26.088041 (+2988437us) ts_tablet_manager.cc:1151] Deleting tablet 0266cef4e2034303978d7456c47ea98c
1024 19:48:26.088048 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.270260 (+182212us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.348555 (+ 78295us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.365554 (+ 16999us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2923708}
I20251024 19:48:26.366029 1776 tablet_replica.cc:333] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:26.366153 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 0266cef4e2034303978d7456c47ea98c (table default.Table_74e6a6d8b7a34049994d41a379a9a9f0 [id=7f87fee0587e4cb5a4f6f5afb3238334]) successfully deleted
I20251024 19:48:26.366987 1776 raft_consensus.cc:2243] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:26.367316 1776 pending_rounds.cc:70] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:26.367378 1776 pending_rounds.cc:77] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429404944805888 op_type: NO_OP noop_request { }
I20251024 19:48:26.367467 1776 raft_consensus.cc:2889] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:26.367553 1776 raft_consensus.cc:2272] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.368392 1776 ts_tablet_manager.cc:1916] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.366428 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92d78642772d44179e67d75c51b9da4e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:26.370291 1444 leader_election.cc:343] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.370427 1444 leader_election.cc:304] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:26.370945 3538 raft_consensus.cc:2749] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:26.373054 1775 ts_tablet_manager.cc:1929] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.373224 1775 log.cc:1199] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/38818c7997c84cedad5f9b09eef77105
I20251024 19:48:26.375082 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet c5cbce9e4b61451ba7f458df922be759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:26.385959 1775 ts_tablet_manager.cc:1950] T 38818c7997c84cedad5f9b09eef77105 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:26.387109 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1029) took 5872 ms. Trace:
I20251024 19:48:26.388094 1775 rpcz_store.cc:276] 1024 19:48:20.514495 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.514579 (+ 84us) service_pool.cc:225] Handling call
1024 19:48:26.274466 (+5759887us) ts_tablet_manager.cc:1151] Deleting tablet 38818c7997c84cedad5f9b09eef77105
1024 19:48:26.298367 (+ 23901us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.333550 (+ 35183us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.372946 (+ 39396us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.387085 (+ 14139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5712361}
W20251024 19:48:26.388339 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04dfa03ab59e4891ad5edd4003b9ceb8 consensus state has no leader
I20251024 19:48:26.388653 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet 2cbeaac19e834ecf98e5b8afc952b9da from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:26.388978 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1030) took 5854 ms. Trace:
I20251024 19:48:26.389053 1775 rpcz_store.cc:276] 1024 19:48:20.534661 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.534733 (+ 72us) service_pool.cc:225] Handling call
1024 19:48:26.388842 (+5854109us) ts_tablet_manager.cc:1151] Deleting tablet a2cc3f3d02324c1e867b8fc159ad6d20
1024 19:48:26.388851 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.388958 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5823451}
W20251024 19:48:26.389361 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92d78642772d44179e67d75c51b9da4e consensus state has no leader
I20251024 19:48:26.389457 1775 tablet_replica.cc:333] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:26.389724 1775 raft_consensus.cc:2243] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.389886 1775 raft_consensus.cc:2272] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.390427 1775 ts_tablet_manager.cc:1916] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.394622 3510 tablet.cc:1722] T b632acc1348e49f8a1eea019c007780c P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251024 19:48:26.397929 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37de0de36bae403287da13caf065ed12 has no consensus state
I20251024 19:48:26.406615 1766 tablet_bootstrap.cc:492] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:26.406781 1766 ts_tablet_manager.cc:1403] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.434s user 0.035s sys 0.090s
I20251024 19:48:26.407544 1766 raft_consensus.cc:359] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:26.407720 1766 raft_consensus.cc:385] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.407776 1766 raft_consensus.cc:740] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.407909 1767 tablet_bootstrap.cc:492] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:26.408047 1767 ts_tablet_manager.cc:1403] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.096s user 0.001s sys 0.002s
I20251024 19:48:26.408001 1766 consensus_queue.cc:260] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:26.408515 1766 ts_tablet_manager.cc:1434] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:26.408787 1766 tablet_bootstrap.cc:492] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:26.410071 1767 raft_consensus.cc:359] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.410233 1767 raft_consensus.cc:385] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.410280 1767 raft_consensus.cc:740] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.410411 1766 tablet_bootstrap.cc:654] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.410445 1767 consensus_queue.cc:260] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:26.413024 1586 consensus_peers.cc:597] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38818c7997c84cedad5f9b09eef77105. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:26.414264 1369 raft_consensus.cc:3060] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.395941 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 38818c7997c84cedad5f9b09eef77105 (table default.Table_d7a9ef5436984a1b83dbeac77c302549 [id=4c4868b6a38046faaf24f65fd00420cb]) successfully deleted
W20251024 19:48:26.417261 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet a2cc3f3d02324c1e867b8fc159ad6d20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2cc3f3d02324c1e867b8fc159ad6d20 already in progress: creating tablet
I20251024 19:48:26.413395 1767 ts_tablet_manager.cc:1434] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:48:26.419943 1767 tablet_bootstrap.cc:492] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:26.420161 1476 tablet_service.cc:1552] Processing DeleteTablet for tablet b22b58404b8b441ba97ef536885a66e0 with delete_type TABLET_DATA_DELETED (replaced by tablet f37a01dc3d5f458d81605f2c41fa27dd) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:26.420989 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e6f8764e9ae472b9ac35a032182aed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:26.424744 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2e8975d63d8c473785779f368a2d2601"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:26.425312 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e8975d63d8c473785779f368a2d2601 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:26.428376 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e5c532eb7fc241039c7e06e4757ea011" } new_table_name: "default.Table_d0930167e31244e789c97b3134c4b7a6" modify_external_catalogs: true
I20251024 19:48:26.429658 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1f4e225434e5437e8119ca0959092f74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:26.430125 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f4e225434e5437e8119ca0959092f74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:26.432696 1766 tablet_bootstrap.cc:492] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:26.432864 1766 ts_tablet_manager.cc:1403] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20251024 19:48:26.432806 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_510bc2b3752b42b09ebef315e1e79a48" } new_table_name: "default.Table_a283640a663643fb9fb94dd62eeaf9de" modify_external_catalogs: true
I20251024 19:48:26.438191 1617 tablet_service.cc:1505] Processing CreateTablet for tablet e2e7a7bf8e5044f98bab85bacb99650e (DEFAULT_TABLE table=default.Table_6b368f6653044b0bb1f7062a5ffa4209 [id=47561a2c26f2405580b9ba0dd36b2d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.438609 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2e7a7bf8e5044f98bab85bacb99650e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.445113 1766 raft_consensus.cc:359] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:26.445295 1766 raft_consensus.cc:385] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.445343 1766 raft_consensus.cc:740] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.447314 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 1af368f347774270bca1a37c65d994ff
I20251024 19:48:26.448073 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 16a8c772ac404ae2b2f86d948a0fcd2f
I20251024 19:48:26.448128 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 1af368f347774270bca1a37c65d994ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:26.448534 1766 consensus_queue.cc:260] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:26.449229 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a05a69bd7d1d4bf291d7cda76658602e consensus state has no leader
I20251024 19:48:26.448755 1189 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_361840e201d943c18014ec71a255cc9c [id=dc00ee30038e49fb9f86499cfb985a0e] alter complete (version 2)
I20251024 19:48:26.453101 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 16a8c772ac404ae2b2f86d948a0fcd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:26.453202 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 73278db2097d4001a538d62dacb97c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:26.453718 1766 ts_tablet_manager.cc:1434] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251024 19:48:26.453943 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 62d307da2a8142e187e603a6e3bd9c58 (DEFAULT_TABLE table=default.Table_430f5855fcbc49a39009fcc1f6bc115f [id=ee9bc7e8367f4a8ba05b00acb0b21891]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.454376 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d307da2a8142e187e603a6e3bd9c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.456113 1766 tablet_bootstrap.cc:492] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:26.463801 1339 tablet_service.cc:1505] Processing CreateTablet for tablet e2e7a7bf8e5044f98bab85bacb99650e (DEFAULT_TABLE table=default.Table_6b368f6653044b0bb1f7062a5ffa4209 [id=47561a2c26f2405580b9ba0dd36b2d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.463964 1767 tablet_bootstrap.cc:654] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.464259 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2e7a7bf8e5044f98bab85bacb99650e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.465010 3528 raft_consensus.cc:493] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:26.465133 3528 raft_consensus.cc:515] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.465698 3528 leader_election.cc:290] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:26.479386 1766 tablet_bootstrap.cc:654] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.485553 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 1af368f347774270bca1a37c65d994ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:26.487390 3538 raft_consensus.cc:493] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:26.487547 3538 raft_consensus.cc:515] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:26.488269 3538 leader_election.cc:290] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:26.488854 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "780442b310e6409498e6bbe4256280a4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:26.489377 1447 leader_election.cc:343] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.489643 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_05d774ae8eb94b3790d1e74b633c8415" } modify_external_catalogs: true
I20251024 19:48:26.489835 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_05d774ae8eb94b3790d1e74b633c8415" } modify_external_catalogs: true
I20251024 19:48:26.490664 1766 tablet_bootstrap.cc:492] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:26.490790 1766 ts_tablet_manager.cc:1403] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20251024 19:48:26.493777 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "780442b310e6409498e6bbe4256280a4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:26.494330 1444 leader_election.cc:343] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.494453 1444 leader_election.cc:304] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:26.494933 3538 raft_consensus.cc:2749] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:26.496881 3360 consensus_queue.cc:1048] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:26.497364 3509 tablet.cc:1722] T 10891ff5bf0d4b93bcab9faa5f7346f1 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:26.498870 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3cdd85a3ca7545eea953e149a6c0d572
I20251024 19:48:26.499151 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 1af368f347774270bca1a37c65d994ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:26.499859 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 16a8c772ac404ae2b2f86d948a0fcd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:26.500090 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 16a8c772ac404ae2b2f86d948a0fcd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:26.500635 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet da7d38b63708427982591da1014aca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:26.500926 1774 tablet_replica.cc:333] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:26.500974 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 4100abc9ff12471e9a4959b6b24f7e99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:25 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:26.501116 1774 raft_consensus.cc:2243] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.501264 1774 raft_consensus.cc:2272] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.502048 1479 tablet_service.cc:1505] Processing CreateTablet for tablet e2e7a7bf8e5044f98bab85bacb99650e (DEFAULT_TABLE table=default.Table_6b368f6653044b0bb1f7062a5ffa4209 [id=47561a2c26f2405580b9ba0dd36b2d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.502115 1774 ts_tablet_manager.cc:1916] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.502528 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 62d307da2a8142e187e603a6e3bd9c58 (DEFAULT_TABLE table=default.Table_430f5855fcbc49a39009fcc1f6bc115f [id=ee9bc7e8367f4a8ba05b00acb0b21891]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.502769 1346 tablet_service.cc:1505] Processing CreateTablet for tablet fa7e31164e8c4dd785610cf5cc5548a4 (DEFAULT_TABLE table=default.Table_05dbfb26c73a4c5aaff27e72dccdf03f [id=0a8f84f1dd4b4dcaabe5bd38302574d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.503116 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa7e31164e8c4dd785610cf5cc5548a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.503562 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 62d307da2a8142e187e603a6e3bd9c58 (DEFAULT_TABLE table=default.Table_430f5855fcbc49a39009fcc1f6bc115f [id=ee9bc7e8367f4a8ba05b00acb0b21891]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.504040 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d307da2a8142e187e603a6e3bd9c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.505999 3524 tablet.cc:1722] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:26.510205 3510 tablet.cc:1722] T 10891ff5bf0d4b93bcab9faa5f7346f1 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:26.511332 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 3e8d7906c9d3403da23d972cf72d9536 (DEFAULT_TABLE table=default.Table_5419e516d52e46228d53bc4afcfe4332 [id=c6160eef08a24bf19ce864007b7d5220]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.511483 1774 ts_tablet_manager.cc:1929] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:26.511608 1774 log.cc:1199] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/da7d38b63708427982591da1014aca81
I20251024 19:48:26.511729 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e8d7906c9d3403da23d972cf72d9536. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.512018 1774 ts_tablet_manager.cc:1950] T da7d38b63708427982591da1014aca81 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:26.513092 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 9014b9ea54f2472c94f35f01de768b47 (DEFAULT_TABLE table=default.Table_eee1b7268a6147b1b8cc5ec4a8af199a [id=5b0ddb1a7c04490e8740941dacb13d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.513409 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9014b9ea54f2472c94f35f01de768b47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.513670 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet 7be9290daabb404f955f05922d897fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:26.513738 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cdd85a3ca7545eea953e149a6c0d572 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:26.513820 1774 tablet_replica.cc:333] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:26.513657 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 2ea938ebe84f496e92a2baa226fb1205 (DEFAULT_TABLE table=default.Table_0dea9bad168942498e4ebfe3bf1eebd3 [id=3261bc8d33ef45799692730136b25e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.514027 1774 raft_consensus.cc:2243] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.514201 1774 raft_consensus.cc:2272] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.514341 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea938ebe84f496e92a2baa226fb1205. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.513657 1623 tablet_service.cc:1505] Processing CreateTablet for tablet fa7e31164e8c4dd785610cf5cc5548a4 (DEFAULT_TABLE table=default.Table_05dbfb26c73a4c5aaff27e72dccdf03f [id=0a8f84f1dd4b4dcaabe5bd38302574d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.516834 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7be9290daabb404f955f05922d897fbf" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:26.517088 1509 raft_consensus.cc:2468] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:26.518424 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7be9290daabb404f955f05922d897fbf" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:26.522028 1177 consensus_peers.cc:597] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c -> Peer 82e82847f8904b59b03f446b7fce6233 (127.0.252.253:41375): Couldn't send request to peer 82e82847f8904b59b03f446b7fce6233. Status: Network error: Client connection negotiation failed: client connection to 127.0.252.253:41375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251024 19:48:26.523612 1774 ts_tablet_manager.cc:1916] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:26.519117 1586 leader_election.cc:343] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.524297 1776 ts_tablet_manager.cc:1929] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.529645 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa7e31164e8c4dd785610cf5cc5548a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.529655 1776 log.cc:1199] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/65424b00ab9c454f9584048caba83e6a
I20251024 19:48:26.529731 1586 leader_election.cc:304] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:26.530051 1776 ts_tablet_manager.cc:1950] T 65424b00ab9c454f9584048caba83e6a P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:26.530380 3360 raft_consensus.cc:2804] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:26.530462 3360 raft_consensus.cc:493] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:26.530535 3360 raft_consensus.cc:3060] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.499713 1766 raft_consensus.cc:359] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.531280 1766 raft_consensus.cc:385] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.531350 1766 raft_consensus.cc:740] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.531580 1766 consensus_queue.cc:260] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.532058 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1104) took 3432 ms. Trace:
I20251024 19:48:26.532147 1766 ts_tablet_manager.cc:1434] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.041s user 0.001s sys 0.000s
I20251024 19:48:26.532181 1776 rpcz_store.cc:276] 1024 19:48:23.099896 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.099942 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:26.365972 (+3266030us) ts_tablet_manager.cc:1151] Deleting tablet 65424b00ab9c454f9584048caba83e6a
1024 19:48:26.365982 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.422460 (+ 56478us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.524198 (+101738us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.532037 (+ 7839us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3201543}
I20251024 19:48:26.532438 1766 tablet_bootstrap.cc:492] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:26.532852 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 65424b00ab9c454f9584048caba83e6a (table default.Table_e6ec362938cc478d9b9abee36f34f5c5 [id=d2e2211a2cd8457aab95305acead0b3b]) successfully deleted
I20251024 19:48:26.533260 3360 raft_consensus.cc:515] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.535997 3360 leader_election.cc:290] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:26.536520 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7be9290daabb404f955f05922d897fbf" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:26.536693 1510 raft_consensus.cc:3060] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.536736 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 475897d75b6646f399b49de7521c03aa (DEFAULT_TABLE table=default.Table_d1d3c977f9fe4fdfa9f638fa7ea69751 [id=fa7e172e74104ba5883cf33b47d612fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.536834 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7be9290daabb404f955f05922d897fbf" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:26.537125 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475897d75b6646f399b49de7521c03aa. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:26.537467 1586 leader_election.cc:343] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.533262 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1111) took 3269 ms. Trace:
I20251024 19:48:26.538450 1776 rpcz_store.cc:276] 1024 19:48:23.263619 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.263733 (+ 114us) service_pool.cc:225] Handling call
1024 19:48:26.533128 (+3269395us) ts_tablet_manager.cc:1151] Deleting tablet cb1f850b3e104707a53d540ad5752d4d
1024 19:48:26.533140 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.533246 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3268626}
W20251024 19:48:26.538892 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet cb1f850b3e104707a53d540ad5752d4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb1f850b3e104707a53d540ad5752d4d already in progress: creating tablet
I20251024 19:48:26.539707 1510 raft_consensus.cc:2468] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:26.540094 1586 leader_election.cc:304] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:26.540477 3360 raft_consensus.cc:2804] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:26.540737 3360 raft_consensus.cc:697] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:26.541157 3360 consensus_queue.cc:237] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.543318 1624 tablet_service.cc:1505] Processing CreateTablet for tablet d319e48a015c4a4cb6937b8048a3582c (DEFAULT_TABLE table=default.Table_b48023645eb94ba4a0341afe616222b6 [id=0a4f143c1a544b9cab371ea2c78eb670]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.543725 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d319e48a015c4a4cb6937b8048a3582c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.544026 3404 raft_consensus.cc:493] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:26.544144 3404 raft_consensus.cc:515] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.544624 3404 leader_election.cc:290] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:26.545230 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73278db2097d4001a538d62dacb97c9b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:26.545712 1586 leader_election.cc:343] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.547065 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1113) took 3242 ms. Trace:
I20251024 19:48:26.547695 1776 rpcz_store.cc:276] 1024 19:48:23.305077 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.313041 (+ 7964us) service_pool.cc:225] Handling call
1024 19:48:26.546938 (+3233897us) ts_tablet_manager.cc:1151] Deleting tablet b24a3a820c9b496dbf6955b399c6ea8c
1024 19:48:26.546947 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.547044 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3233494}
I20251024 19:48:26.548934 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73278db2097d4001a538d62dacb97c9b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:26.549165 1510 raft_consensus.cc:2468] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:26.550195 1191 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_18e7285b7df44399ab3be1dfc86eb91f [id=14bda7f9324f44aa9cb87a5f129039be] alter complete (version 2)
I20251024 19:48:26.550729 1768 tablet_bootstrap.cc:492] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:26.550959 1768 ts_tablet_manager.cc:1403] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.201s user 0.009s sys 0.006s
I20251024 19:48:26.551676 1768 raft_consensus.cc:359] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.551923 1768 raft_consensus.cc:385] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.551990 1768 raft_consensus.cc:740] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.549618 1586 leader_election.cc:304] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:26.552201 1768 consensus_queue.cc:260] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.552527 3404 raft_consensus.cc:2804] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:26.552613 3404 raft_consensus.cc:493] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:26.552709 3404 raft_consensus.cc:3060] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:26.554010 1559 tablet.cc:2378] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 19:48:26.555238 3404 raft_consensus.cc:515] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.555773 3404 leader_election.cc:290] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:26.556156 1768 ts_tablet_manager.cc:1434] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251024 19:48:26.556571 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73278db2097d4001a538d62dacb97c9b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:26.557070 1768 tablet_bootstrap.cc:492] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:26.557046 1586 leader_election.cc:343] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.557343 1776 tablet_replica.cc:333] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:26.557565 1776 raft_consensus.cc:2243] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.557878 1766 tablet_bootstrap.cc:654] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:26.557971 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b24a3a820c9b496dbf6955b399c6ea8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b24a3a820c9b496dbf6955b399c6ea8c already in progress: creating tablet
I20251024 19:48:26.558413 1776 raft_consensus.cc:2272] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.558944 1768 tablet_bootstrap.cc:654] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.559087 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73278db2097d4001a538d62dacb97c9b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:26.559322 1510 raft_consensus.cc:3060] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.560034 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2e7a7bf8e5044f98bab85bacb99650e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.560252 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d307da2a8142e187e603a6e3bd9c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.562337 1510 raft_consensus.cc:2468] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:26.564103 1586 leader_election.cc:304] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:26.564713 3404 raft_consensus.cc:2804] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:26.564874 3404 raft_consensus.cc:697] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:26.565156 3404 consensus_queue.cc:237] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.566021 1490 tablet_service.cc:1505] Processing CreateTablet for tablet fa7e31164e8c4dd785610cf5cc5548a4 (DEFAULT_TABLE table=default.Table_05dbfb26c73a4c5aaff27e72dccdf03f [id=0a8f84f1dd4b4dcaabe5bd38302574d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.566398 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa7e31164e8c4dd785610cf5cc5548a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.569890 1624 tablet_service.cc:1505] Processing CreateTablet for tablet e96521c6871e45f08cd9db3190468f3e (DEFAULT_TABLE table=default.Table_b7f521e0feb04a5fa8c1475090117699 [id=87968bfb1a564ba988da40bb5c90c2c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.570361 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e96521c6871e45f08cd9db3190468f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.549882 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 2350cc862586424db3e981656e2da410 (DEFAULT_TABLE table=default.Table_0228067f17b24fc0b45df5fd3f970997 [id=a2e2391f83ab44058b3fa02bf94f11b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.571870 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2350cc862586424db3e981656e2da410. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.575143 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 9014b9ea54f2472c94f35f01de768b47 (DEFAULT_TABLE table=default.Table_eee1b7268a6147b1b8cc5ec4a8af199a [id=5b0ddb1a7c04490e8740941dacb13d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.575628 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9014b9ea54f2472c94f35f01de768b47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.576668 1775 ts_tablet_manager.cc:1929] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.576874 1775 log.cc:1199] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/250f106664be4b5aaa330dbb4e43cb7b
I20251024 19:48:26.577473 1775 ts_tablet_manager.cc:1950] T 250f106664be4b5aaa330dbb4e43cb7b P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:26.578835 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 5c2d6530374442cd9ba6d0080114a680 (DEFAULT_TABLE table=default.Table_713a394d7b9f4e78b25f9569145f4d1d [id=b7f39e671e5749bf8aadce4ac01ad8fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.579267 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c2d6530374442cd9ba6d0080114a680. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.583187 1776 ts_tablet_manager.cc:1916] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.585546 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 2ea938ebe84f496e92a2baa226fb1205 (DEFAULT_TABLE table=default.Table_0dea9bad168942498e4ebfe3bf1eebd3 [id=3261bc8d33ef45799692730136b25e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.586079 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea938ebe84f496e92a2baa226fb1205. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.595141 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cdd85a3ca7545eea953e149a6c0d572 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:26.600376 1191 catalog_manager.cc:5649] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:26.601236 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 0797d14e9fd846788164747523df0e2c (table default.Table_1f866642ce4a4ee885f169c4815bd1fb [id=a07755e138c84fd18c0fe72e81ab758b]). Expected version 1 got 0
I20251024 19:48:26.601233 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 2cbb82d8833d46d7ab2c87c26c99fcbd (DEFAULT_TABLE table=default.Table_9dec6c29bd194b869699fef3b34923c8 [id=8e8f995b59054e7ca05b9a524d5ceb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.601863 1774 ts_tablet_manager.cc:1929] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:26.602069 1774 log.cc:1199] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/4100abc9ff12471e9a4959b6b24f7e99
I20251024 19:48:26.602509 1774 ts_tablet_manager.cc:1950] T 4100abc9ff12471e9a4959b6b24f7e99 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:26.603358 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cbb82d8833d46d7ab2c87c26c99fcbd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:26.608381 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 1af368f347774270bca1a37c65d994ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1af368f347774270bca1a37c65d994ff already in progress: creating tablet
W20251024 19:48:26.608616 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 16a8c772ac404ae2b2f86d948a0fcd2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16a8c772ac404ae2b2f86d948a0fcd2f already in progress: creating tablet
I20251024 19:48:26.608767 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet da7d38b63708427982591da1014aca81 (table default.Table_5744ec49c5e349d79d7473d6ed4154c6 [id=334320a439bf4611ba5b1aa660844023]) successfully deleted
I20251024 19:48:26.609127 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 4100abc9ff12471e9a4959b6b24f7e99 (table default.Table_da24358ca9544a0e805bb937715888f9 [id=0e3c3f6ff360497c93fac944f351702c]) successfully deleted
W20251024 19:48:26.609292 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3cdd85a3ca7545eea953e149a6c0d572 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cdd85a3ca7545eea953e149a6c0d572 already in progress: creating tablet
I20251024 19:48:26.611016 1767 tablet_bootstrap.cc:492] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:26.611196 1767 ts_tablet_manager.cc:1403] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.191s user 0.002s sys 0.001s
I20251024 19:48:26.612033 1767 raft_consensus.cc:359] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.612257 1767 raft_consensus.cc:385] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.612365 1767 raft_consensus.cc:740] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.612658 1767 consensus_queue.cc:260] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.612679 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 3e8d7906c9d3403da23d972cf72d9536 (DEFAULT_TABLE table=default.Table_5419e516d52e46228d53bc4afcfe4332 [id=c6160eef08a24bf19ce864007b7d5220]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.613204 1767 ts_tablet_manager.cc:1434] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:26.613353 1768 tablet_bootstrap.cc:492] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:26.612188 3360 consensus_queue.cc:1048] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:26.613494 1768 ts_tablet_manager.cc:1403] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.057s user 0.000s sys 0.003s
I20251024 19:48:26.614336 1768 raft_consensus.cc:359] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.614526 1768 raft_consensus.cc:385] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.614591 1768 raft_consensus.cc:740] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.613888 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 81d2a1d435d14912a160e96a6e787be5 (DEFAULT_TABLE table=default.Table_0d59c786f9c144b9a5b260a3355773ad [id=22ffe14703384ef1b4ebc2c50ae99959]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.614820 1768 consensus_queue.cc:260] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.615221 1767 tablet_bootstrap.cc:492] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:26.615283 1768 ts_tablet_manager.cc:1434] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:26.615579 1768 tablet_bootstrap.cc:492] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:26.584385 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 2ea938ebe84f496e92a2baa226fb1205 (DEFAULT_TABLE table=default.Table_0dea9bad168942498e4ebfe3bf1eebd3 [id=3261bc8d33ef45799692730136b25e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.617835 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea938ebe84f496e92a2baa226fb1205. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.584725 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 9014b9ea54f2472c94f35f01de768b47 (DEFAULT_TABLE table=default.Table_eee1b7268a6147b1b8cc5ec4a8af199a [id=5b0ddb1a7c04490e8740941dacb13d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.590374 3538 raft_consensus.cc:493] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:26.618011 3538 raft_consensus.cc:515] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:26.618621 3538 leader_election.cc:290] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:26.619272 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9014b9ea54f2472c94f35f01de768b47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.619742 1767 tablet_bootstrap.cc:654] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.620915 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc31e6c25ad14947af20abd766f12e78" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:26.621484 1447 leader_election.cc:343] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.595110 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 3e8d7906c9d3403da23d972cf72d9536 (DEFAULT_TABLE table=default.Table_5419e516d52e46228d53bc4afcfe4332 [id=c6160eef08a24bf19ce864007b7d5220]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.595646 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cdd85a3ca7545eea953e149a6c0d572 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:26.604682 1444 consensus_peers.cc:597] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:26.622057 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e8d7906c9d3403da23d972cf72d9536. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.622484 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc31e6c25ad14947af20abd766f12e78" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:26.623011 1444 leader_election.cc:343] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.623153 1444 leader_election.cc:304] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:26.623653 3538 raft_consensus.cc:2749] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:26.632198 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1031) took 6033 ms. Trace:
I20251024 19:48:26.632377 1775 rpcz_store.cc:276] 1024 19:48:20.598936 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.600268 (+ 1332us) service_pool.cc:225] Handling call
1024 19:48:26.389411 (+5789143us) ts_tablet_manager.cc:1151] Deleting tablet 250f106664be4b5aaa330dbb4e43cb7b
1024 19:48:26.389421 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.560029 (+170608us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.576532 (+ 16503us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.632175 (+ 55643us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5769345}
I20251024 19:48:26.632833 1775 tablet_replica.cc:333] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:26.633127 1775 raft_consensus.cc:2243] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:26.633513 1775 raft_consensus.cc:2272] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.638238 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:26.639955 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d2a1d435d14912a160e96a6e787be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.640995 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_319c59c379f548e0b483ab52e73664eb" } modify_external_catalogs: true
I20251024 19:48:26.641312 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_319c59c379f548e0b483ab52e73664eb" } modify_external_catalogs: true
I20251024 19:48:26.612066 1766 tablet_bootstrap.cc:492] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:26.645771 1766 ts_tablet_manager.cc:1403] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.113s user 0.003s sys 0.000s
I20251024 19:48:26.646552 1776 ts_tablet_manager.cc:1929] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.646648 1766 raft_consensus.cc:359] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:26.646838 1766 raft_consensus.cc:385] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.646914 1766 raft_consensus.cc:740] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.647271 1766 consensus_queue.cc:260] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:26.649137 1766 ts_tablet_manager.cc:1434] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:26.649857 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f041d5ea635b4c3db554dbcbf3b3c50f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:26.650408 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f041d5ea635b4c3db554dbcbf3b3c50f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:26.650519 1766 tablet_bootstrap.cc:492] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:26.654874 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1d23448dda3547c0a40cc1abbbd4ffc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:26.654985 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_34787bfb5ad443ff9c6ef9a773ddb6ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:26.654943 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_1175a8d8c6144a8d9479b68f16c35b05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:26.655426 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d23448dda3547c0a40cc1abbbd4ffc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:26.656098 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5f5eee56374d49cd8a80b28f26f42bac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:26.655409 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1175a8d8c6144a8d9479b68f16c35b05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:26.656584 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f5eee56374d49cd8a80b28f26f42bac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:26.657820 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet 60fbfb1620934034af2acaa39677554f from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:26.658092 1768 tablet_bootstrap.cc:654] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.659528 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet d391bc3b3a144415bd4e6b5fdcc37612
I20251024 19:48:26.660929 1775 ts_tablet_manager.cc:1916] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.661180 1776 log.cc:1199] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/81f8393f78204860bd854f4872099387
I20251024 19:48:26.660260 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet d391bc3b3a144415bd4e6b5fdcc37612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:26.655409 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34787bfb5ad443ff9c6ef9a773ddb6ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:26.660341 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 972f9ca0d2234a5f8d698090a6825bfb (DEFAULT_TABLE table=default.Table_db62205c9d4b431bb48b4b75d0fcf8d1 [id=619e60f84da24d899ab2f69923f459b1]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:26.660465 1586 consensus_peers.cc:597] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 250f106664be4b5aaa330dbb4e43cb7b. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:26.662252 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 972f9ca0d2234a5f8d698090a6825bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.662674 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e8d7906c9d3403da23d972cf72d9536. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.663420 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 2350cc862586424db3e981656e2da410 (DEFAULT_TABLE table=default.Table_0228067f17b24fc0b45df5fd3f970997 [id=a2e2391f83ab44058b3fa02bf94f11b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.664083 1476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2350cc862586424db3e981656e2da410. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.659677 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 47d0bb51ac674a3c900ec5773ebfd317 (DEFAULT_TABLE table=default.Table_da73cd6e87304788ad9da534dbc041a5 [id=cdc3cae49be447528e4d6d387100e8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.667861 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47d0bb51ac674a3c900ec5773ebfd317. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.670229 1776 ts_tablet_manager.cc:1950] T 81f8393f78204860bd854f4872099387 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:26.670951 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_01c58de0bf6845b08ff6fab88827cdec" } modify_external_catalogs: true
I20251024 19:48:26.671177 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_01c58de0bf6845b08ff6fab88827cdec" } modify_external_catalogs: true
I20251024 19:48:26.672166 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5035151c0890403b9dc59e3973c94e76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:26.672761 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5035151c0890403b9dc59e3973c94e76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:26.672868 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d3cbf85ffa894f149214169d2e8c9977"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:26.673354 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3cbf85ffa894f149214169d2e8c9977 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:26.676729 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_286aa7130af04dcfb71514e1f9add870"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:26.676939 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_86e47213e04046e89c4ef6974b4b3630" } new_table_name: "default.Table_dbde3080b2134be694fae5226e2b48de" modify_external_catalogs: true
W20251024 19:48:26.677242 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_286aa7130af04dcfb71514e1f9add870 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:26.680542 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1126) took 3186 ms. Trace:
I20251024 19:48:26.680635 1776 rpcz_store.cc:276] 1024 19:48:23.493845 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.493918 (+ 73us) service_pool.cc:225] Handling call
1024 19:48:26.557295 (+3063377us) ts_tablet_manager.cc:1151] Deleting tablet 81f8393f78204860bd854f4872099387
1024 19:48:26.557305 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.588721 (+ 31416us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.646447 (+ 57726us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.680516 (+ 34069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3063161}
I20251024 19:48:26.681020 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 81f8393f78204860bd854f4872099387 (table default.Table_2d74dcbcd9f249e5b10e1986cc0c0d18 [id=0bd294ee0c8341a6a29ce26caeeceb22]) successfully deleted
W20251024 19:48:26.683357 1586 consensus_peers.cc:597] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:26.683743 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 2350cc862586424db3e981656e2da410 (DEFAULT_TABLE table=default.Table_0228067f17b24fc0b45df5fd3f970997 [id=a2e2391f83ab44058b3fa02bf94f11b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.684187 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2350cc862586424db3e981656e2da410. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.689011 3524 tablet.cc:1722] T 7503b22d9e12481b9af96385b9d76e50 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251024 19:48:26.690330 1767 tablet_bootstrap.cc:492] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:26.690477 3509 tablet.cc:1722] T 7503b22d9e12481b9af96385b9d76e50 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251024 19:48:26.690582 1767 ts_tablet_manager.cc:1403] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.005s
I20251024 19:48:26.691340 1767 raft_consensus.cc:359] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.691576 1767 raft_consensus.cc:385] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.691675 1767 raft_consensus.cc:740] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.691941 1767 consensus_queue.cc:260] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.692581 1767 ts_tablet_manager.cc:1434] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:26.692924 1767 tablet_bootstrap.cc:492] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:26.694511 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1124) took 3280 ms. Trace:
I20251024 19:48:26.694667 1776 rpcz_store.cc:276] 1024 19:48:23.414312 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.414446 (+ 134us) service_pool.cc:225] Handling call
1024 19:48:26.694375 (+3279929us) ts_tablet_manager.cc:1151] Deleting tablet 830c8469331940fc85aaad58c9085e35
1024 19:48:26.694384 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.694490 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3187512}
I20251024 19:48:26.695113 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4dca2888019448ca97c68a8c6dd8396a" } new_table_name: "default.Table_207e09d4c2724721ac0e69fc49ef8dea" modify_external_catalogs: true
W20251024 19:48:26.695233 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 830c8469331940fc85aaad58c9085e35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 830c8469331940fc85aaad58c9085e35 already in progress: creating tablet
I20251024 19:48:26.695652 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1129) took 3185 ms. Trace:
I20251024 19:48:26.695740 1776 rpcz_store.cc:276] 1024 19:48:23.510054 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.582196 (+ 72142us) service_pool.cc:225] Handling call
1024 19:48:26.695534 (+3113338us) ts_tablet_manager.cc:1151] Deleting tablet 2d8b8a1cff3540f7b7f13712425d534e
1024 19:48:26.695544 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.695633 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3089767}
I20251024 19:48:26.696121 1334 tablet_service.cc:1505] Processing CreateTablet for tablet d319e48a015c4a4cb6937b8048a3582c (DEFAULT_TABLE table=default.Table_b48023645eb94ba4a0341afe616222b6 [id=0a4f143c1a544b9cab371ea2c78eb670]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:26.696282 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 2d8b8a1cff3540f7b7f13712425d534e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d8b8a1cff3540f7b7f13712425d534e already in progress: creating tablet
I20251024 19:48:26.696503 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d319e48a015c4a4cb6937b8048a3582c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.696610 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1125) took 3202 ms. Trace:
I20251024 19:48:26.696671 1776 rpcz_store.cc:276] 1024 19:48:23.493710 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.612762 (+119052us) service_pool.cc:225] Handling call
1024 19:48:26.696506 (+3083744us) ts_tablet_manager.cc:1151] Deleting tablet 4846799df5b242ce8f19c034320c8a1d
1024 19:48:26.696516 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.696594 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3083290}
W20251024 19:48:26.697048 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4846799df5b242ce8f19c034320c8a1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4846799df5b242ce8f19c034320c8a1d already in progress: creating tablet
I20251024 19:48:26.697342 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1136) took 2783 ms. Trace:
I20251024 19:48:26.697408 1776 rpcz_store.cc:276] 1024 19:48:23.914085 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.914159 (+ 74us) service_pool.cc:225] Handling call
1024 19:48:26.697247 (+2783088us) ts_tablet_manager.cc:1151] Deleting tablet 03d8ab62b47f474198841ea14a3b7b00
1024 19:48:26.697256 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.697327 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2782758}
W20251024 19:48:26.697768 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 03d8ab62b47f474198841ea14a3b7b00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03d8ab62b47f474198841ea14a3b7b00 already in progress: creating tablet
I20251024 19:48:26.698787 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1137) took 2671 ms. Trace:
I20251024 19:48:26.699615 1776 rpcz_store.cc:276] 1024 19:48:24.027142 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.027244 (+ 102us) service_pool.cc:225] Handling call
1024 19:48:26.698069 (+2670825us) ts_tablet_manager.cc:1151] Deleting tablet 0a5034103c1e4e039df7fc9f0ab65d0c
1024 19:48:26.698688 (+ 619us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.698773 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2670631}
W20251024 19:48:26.700002 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 0a5034103c1e4e039df7fc9f0ab65d0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a5034103c1e4e039df7fc9f0ab65d0c already in progress: creating tablet
I20251024 19:48:26.700516 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1140) took 2555 ms. Trace:
I20251024 19:48:26.700587 1776 rpcz_store.cc:276] 1024 19:48:24.144855 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.144925 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:26.700250 (+2555325us) ts_tablet_manager.cc:1151] Deleting tablet 33d42f4c818340cda4ae615132fcf03f
1024 19:48:26.700261 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.700499 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2555092}
W20251024 19:48:26.700945 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 33d42f4c818340cda4ae615132fcf03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33d42f4c818340cda4ae615132fcf03f already in progress: creating tablet
I20251024 19:48:26.701264 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1172) took 2105 ms. Trace:
I20251024 19:48:26.701372 1776 rpcz_store.cc:276] 1024 19:48:24.595628 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.595674 (+ 46us) service_pool.cc:225] Handling call
1024 19:48:26.701162 (+2105488us) ts_tablet_manager.cc:1151] Deleting tablet 91bb076b4b0d4b04955043248460d04c
1024 19:48:26.701173 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.701247 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2105333}
W20251024 19:48:26.701686 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 91bb076b4b0d4b04955043248460d04c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91bb076b4b0d4b04955043248460d04c already in progress: creating tablet
I20251024 19:48:26.704021 1768 tablet_bootstrap.cc:492] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:26.704165 1768 ts_tablet_manager.cc:1403] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.000s
I20251024 19:48:26.705241 1768 raft_consensus.cc:359] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.705426 1768 raft_consensus.cc:385] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.705492 1768 raft_consensus.cc:740] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.705765 1768 consensus_queue.cc:260] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.694617 1767 tablet_bootstrap.cc:654] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.708262 1768 ts_tablet_manager.cc:1434] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:26.709967 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 475897d75b6646f399b49de7521c03aa (DEFAULT_TABLE table=default.Table_d1d3c977f9fe4fdfa9f638fa7ea69751 [id=fa7e172e74104ba5883cf33b47d612fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.710031 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 23d4dd0b42d343cb9354ac698a8c761f with delete_type TABLET_DATA_DELETED (replaced by tablet fce7de9523fb4878a58aa8aef0c6b182) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:26.709312 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 475897d75b6646f399b49de7521c03aa (DEFAULT_TABLE table=default.Table_d1d3c977f9fe4fdfa9f638fa7ea69751 [id=fa7e172e74104ba5883cf33b47d612fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.710400 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475897d75b6646f399b49de7521c03aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.710546 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475897d75b6646f399b49de7521c03aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.717905 1490 tablet_service.cc:1505] Processing CreateTablet for tablet d319e48a015c4a4cb6937b8048a3582c (DEFAULT_TABLE table=default.Table_b48023645eb94ba4a0341afe616222b6 [id=0a4f143c1a544b9cab371ea2c78eb670]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.718389 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d319e48a015c4a4cb6937b8048a3582c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.718369 1768 tablet_bootstrap.cc:492] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:26.718843 1774 tablet_replica.cc:333] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:26.719053 1774 raft_consensus.cc:2243] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.719208 1774 raft_consensus.cc:2272] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.720055 1774 ts_tablet_manager.cc:1916] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.722272 1350 tablet_service.cc:1505] Processing CreateTablet for tablet e96521c6871e45f08cd9db3190468f3e (DEFAULT_TABLE table=default.Table_b7f521e0feb04a5fa8c1475090117699 [id=87968bfb1a564ba988da40bb5c90c2c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.723366 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 5c2d6530374442cd9ba6d0080114a680 (DEFAULT_TABLE table=default.Table_713a394d7b9f4e78b25f9569145f4d1d [id=b7f39e671e5749bf8aadce4ac01ad8fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.724084 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 5c2d6530374442cd9ba6d0080114a680 (DEFAULT_TABLE table=default.Table_713a394d7b9f4e78b25f9569145f4d1d [id=b7f39e671e5749bf8aadce4ac01ad8fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.724406 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c2d6530374442cd9ba6d0080114a680. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.720366 1766 tablet_bootstrap.cc:654] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:26.725045 3466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 19:48:26.725414 3466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 19:48:26.725520 3466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251024 19:48:26.725891 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1171) took 2130 ms. Trace:
I20251024 19:48:26.725978 1776 rpcz_store.cc:276] 1024 19:48:24.595500 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.596108 (+ 608us) service_pool.cc:225] Handling call
1024 19:48:26.725731 (+2129623us) ts_tablet_manager.cc:1151] Deleting tablet 4b4904080b7a449fb92e773032808533
1024 19:48:26.725744 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.725865 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2129401}
W20251024 19:48:26.726436 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 4b4904080b7a449fb92e773032808533 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b4904080b7a449fb92e773032808533 already in progress: creating tablet
I20251024 19:48:26.726598 1774 ts_tablet_manager.cc:1929] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:26.726702 1774 log.cc:1199] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/23d4dd0b42d343cb9354ac698a8c761f
I20251024 19:48:26.726867 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1173) took 2126 ms. Trace:
I20251024 19:48:26.727012 1774 ts_tablet_manager.cc:1950] T 23d4dd0b42d343cb9354ac698a8c761f P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:26.726939 1776 rpcz_store.cc:276] 1024 19:48:24.600207 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.600254 (+ 47us) service_pool.cc:225] Handling call
1024 19:48:26.726760 (+2126506us) ts_tablet_manager.cc:1151] Deleting tablet 9229fa6bf13441fa894e7bb7f78d7038
1024 19:48:26.726770 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.726851 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2126293}
W20251024 19:48:26.727290 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9229fa6bf13441fa894e7bb7f78d7038 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9229fa6bf13441fa894e7bb7f78d7038 already in progress: creating tablet
I20251024 19:48:26.727591 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1174) took 2118 ms. Trace:
I20251024 19:48:26.727658 1776 rpcz_store.cc:276] 1024 19:48:24.608657 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.608834 (+ 177us) service_pool.cc:225] Handling call
1024 19:48:26.727509 (+2118675us) ts_tablet_manager.cc:1151] Deleting tablet 9a1031d785b646b095ece9be666d2a77
1024 19:48:26.727518 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.727574 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2118498}
W20251024 19:48:26.728035 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 9a1031d785b646b095ece9be666d2a77 because the tablet was not found. No further retry: Not found: Tablet not found: 9a1031d785b646b095ece9be666d2a77
I20251024 19:48:26.728295 1776 tablet_replica.cc:333] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:26.728472 1776 raft_consensus.cc:2243] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:26.728818 1768 tablet_bootstrap.cc:654] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.728848 1776 raft_consensus.cc:2272] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.729406 1776 ts_tablet_manager.cc:1916] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.730826 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e96521c6871e45f08cd9db3190468f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.731051 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c2d6530374442cd9ba6d0080114a680. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.734704 1479 tablet_service.cc:1505] Processing CreateTablet for tablet e96521c6871e45f08cd9db3190468f3e (DEFAULT_TABLE table=default.Table_b7f521e0feb04a5fa8c1475090117699 [id=87968bfb1a564ba988da40bb5c90c2c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.737025 1775 ts_tablet_manager.cc:1929] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.737345 1775 log.cc:1199] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/2e7b3288182744969e217d4f174fe4ea
I20251024 19:48:26.737848 1775 ts_tablet_manager.cc:1950] T 2e7b3288182744969e217d4f174fe4ea P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:26.738857 1767 tablet_bootstrap.cc:492] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:26.739051 1767 ts_tablet_manager.cc:1403] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20251024 19:48:26.739825 1767 raft_consensus.cc:359] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.740056 1767 raft_consensus.cc:385] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.740159 1767 raft_consensus.cc:740] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.740442 1767 consensus_queue.cc:260] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.740984 1767 ts_tablet_manager.cc:1434] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:26.741340 1767 tablet_bootstrap.cc:492] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:26.742717 3404 consensus_queue.cc:1048] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:26.742874 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 81d2a1d435d14912a160e96a6e787be5 (DEFAULT_TABLE table=default.Table_0d59c786f9c144b9a5b260a3355773ad [id=22ffe14703384ef1b4ebc2c50ae99959]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.744835 1767 tablet_bootstrap.cc:654] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.737262 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e96521c6871e45f08cd9db3190468f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.747011 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 2cbb82d8833d46d7ab2c87c26c99fcbd (DEFAULT_TABLE table=default.Table_9dec6c29bd194b869699fef3b34923c8 [id=8e8f995b59054e7ca05b9a524d5ceb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.747082 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 7be9290daabb404f955f05922d897fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:26.748477 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 972f9ca0d2234a5f8d698090a6825bfb (DEFAULT_TABLE table=default.Table_db62205c9d4b431bb48b4b75d0fcf8d1 [id=619e60f84da24d899ab2f69923f459b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.748951 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 47d0bb51ac674a3c900ec5773ebfd317 (DEFAULT_TABLE table=default.Table_da73cd6e87304788ad9da534dbc041a5 [id=cdc3cae49be447528e4d6d387100e8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.749526 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet d391bc3b3a144415bd4e6b5fdcc37612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:26.750801 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 250f106664be4b5aaa330dbb4e43cb7b (table default.Table_082b885410b34b1cb2f154cdbd9700a8 [id=963d13a1679345489335e30a9aa349b1]) successfully deleted
I20251024 19:48:26.746117 3510 tablet.cc:1722] T 7503b22d9e12481b9af96385b9d76e50 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251024 19:48:26.747514 1480 tablet_service.cc:1505] Processing CreateTablet for tablet 2cbb82d8833d46d7ab2c87c26c99fcbd (DEFAULT_TABLE table=default.Table_9dec6c29bd194b869699fef3b34923c8 [id=8e8f995b59054e7ca05b9a524d5ceb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.747596 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet ebcc09d8e029451d8fb6b6601ad16087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:26.757839 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cbb82d8833d46d7ab2c87c26c99fcbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.748795 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 972f9ca0d2234a5f8d698090a6825bfb (DEFAULT_TABLE table=default.Table_db62205c9d4b431bb48b4b75d0fcf8d1 [id=619e60f84da24d899ab2f69923f459b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.761059 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 972f9ca0d2234a5f8d698090a6825bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.743899 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 81d2a1d435d14912a160e96a6e787be5 (DEFAULT_TABLE table=default.Table_0d59c786f9c144b9a5b260a3355773ad [id=22ffe14703384ef1b4ebc2c50ae99959]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.762295 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d2a1d435d14912a160e96a6e787be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.749171 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 47d0bb51ac674a3c900ec5773ebfd317 (DEFAULT_TABLE table=default.Table_da73cd6e87304788ad9da534dbc041a5 [id=cdc3cae49be447528e4d6d387100e8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:26.766111 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47d0bb51ac674a3c900ec5773ebfd317. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.750366 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet d391bc3b3a144415bd4e6b5fdcc37612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:26.774701 1776 ts_tablet_manager.cc:1929] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.786794 1776 log.cc:1199] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/250f106664be4b5aaa330dbb4e43cb7b
I20251024 19:48:26.790408 1776 ts_tablet_manager.cc:1950] T 250f106664be4b5aaa330dbb4e43cb7b P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:26.792011 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1175) took 2183 ms. Trace:
I20251024 19:48:26.738969 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1032) took 5850 ms. Trace:
I20251024 19:48:26.793476 1775 rpcz_store.cc:276] 1024 19:48:20.888277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.888339 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:26.632788 (+5744449us) ts_tablet_manager.cc:1151] Deleting tablet 2e7b3288182744969e217d4f174fe4ea
1024 19:48:26.632798 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.664425 (+ 31627us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.736930 (+ 72505us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.738948 (+ 2018us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5744208}
I20251024 19:48:26.794121 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 2e7b3288182744969e217d4f174fe4ea (table default.Table_c4882268578941ccaf2c28be3074fdd2 [id=0b00aa7d2eaa4b1a967952b9a484fd42]) successfully deleted
I20251024 19:48:26.795223 1776 rpcz_store.cc:276] 1024 19:48:24.608775 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.616699 (+ 7924us) service_pool.cc:225] Handling call
1024 19:48:26.728252 (+2111553us) ts_tablet_manager.cc:1151] Deleting tablet 250f106664be4b5aaa330dbb4e43cb7b
1024 19:48:26.728260 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.732588 (+ 4328us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.774586 (+ 41998us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.791988 (+ 17402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2111371}
I20251024 19:48:26.796006 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 250f106664be4b5aaa330dbb4e43cb7b (table default.Table_082b885410b34b1cb2f154cdbd9700a8 [id=963d13a1679345489335e30a9aa349b1]) successfully deleted
I20251024 19:48:26.796501 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1176) took 2143 ms. Trace:
I20251024 19:48:26.796931 1776 rpcz_store.cc:276] 1024 19:48:24.652678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.659190 (+ 6512us) service_pool.cc:225] Handling call
1024 19:48:26.796351 (+2137161us) ts_tablet_manager.cc:1151] Deleting tablet 95e11ed28aab4e809e8fc1615f21744c
1024 19:48:26.796361 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.796482 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2136902}
I20251024 19:48:26.798039 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1182) took 1991 ms. Trace:
I20251024 19:48:26.798740 1775 tablet_replica.cc:333] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:26.799026 1775 raft_consensus.cc:2243] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:26.801323 1776 rpcz_store.cc:276] 1024 19:48:24.806377 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.806467 (+ 90us) service_pool.cc:225] Handling call
1024 19:48:26.797920 (+1991453us) ts_tablet_manager.cc:1151] Deleting tablet 913fee0ca74e4cf99d6697ad5c38f4eb
1024 19:48:26.797931 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.798020 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1976169}
W20251024 19:48:26.801983 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 913fee0ca74e4cf99d6697ad5c38f4eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 913fee0ca74e4cf99d6697ad5c38f4eb already in progress: creating tablet
I20251024 19:48:26.802361 1776 tablet_replica.cc:333] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:26.804420 1776 raft_consensus.cc:2243] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.804826 1776 raft_consensus.cc:2272] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.805615 1776 ts_tablet_manager.cc:1916] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.808758 1775 raft_consensus.cc:2272] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.814145 1775 ts_tablet_manager.cc:1916] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:26.822945 1586 consensus_peers.cc:597] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1d26e02ad2945b39cf27738ed5b7cf4. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:26.825680 1178 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 (table default.Table_575ba6a1b4c64e77aadef06e3084a52a [id=b54040c991354937a9ba3afdd9a75470], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f5868ac1f12e4ef5b4a3002a0d9ac4b9 consensus state has no leader
I20251024 19:48:26.830420 1768 tablet_bootstrap.cc:492] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:26.830607 1768 ts_tablet_manager.cc:1403] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.113s user 0.003s sys 0.001s
I20251024 19:48:26.831501 1768 raft_consensus.cc:359] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.831673 1768 raft_consensus.cc:385] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.831727 1768 raft_consensus.cc:740] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.831943 1768 consensus_queue.cc:260] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.832403 1768 ts_tablet_manager.cc:1434] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:26.836602 1775 ts_tablet_manager.cc:1929] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.838029 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 23d4dd0b42d343cb9354ac698a8c761f (table default.Table_628b2afa9401426980be0eae5dffd917 [id=5b96ec1514ae41e3a0103e4b381aab61]) successfully deleted
I20251024 19:48:26.837533 1774 tablet_replica.cc:333] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:26.838485 1774 raft_consensus.cc:2243] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.838698 1774 raft_consensus.cc:2272] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.839478 1774 ts_tablet_manager.cc:1916] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.840090 1768 tablet_bootstrap.cc:492] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:26.848668 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d2a1d435d14912a160e96a6e787be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.855484 1774 ts_tablet_manager.cc:1929] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:26.855705 1774 log.cc:1199] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/7be9290daabb404f955f05922d897fbf
W20251024 19:48:26.855713 3466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20251024 19:48:26.856110 1774 ts_tablet_manager.cc:1950] T 7be9290daabb404f955f05922d897fbf P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:26.856761 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 7be9290daabb404f955f05922d897fbf (table default.Table_65097482f924409b87d7f4dcb5029849 [id=867c6e52ff2f4eb3a45495738770709d]) successfully deleted
W20251024 19:48:26.857511 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet d391bc3b3a144415bd4e6b5fdcc37612 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d391bc3b3a144415bd4e6b5fdcc37612 already in progress: creating tablet
I20251024 19:48:26.857690 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47d0bb51ac674a3c900ec5773ebfd317. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.858103 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 972f9ca0d2234a5f8d698090a6825bfb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:26.865258 3466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20251024 19:48:26.865244 1775 log.cc:1199] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/23d4dd0b42d343cb9354ac698a8c761f
W20251024 19:48:26.865401 3466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20251024 19:48:26.865715 1775 ts_tablet_manager.cc:1950] T 23d4dd0b42d343cb9354ac698a8c761f P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
W20251024 19:48:26.865866 3466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 19:48:26.866564 3573 raft_consensus.cc:493] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:26.866714 3573 raft_consensus.cc:515] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.867364 3573 leader_election.cc:290] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:26.868041 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e38ef2ac19d14f9c91fa8abe86cc7987" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:26.868577 1586 leader_election.cc:343] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:26.866320 1444 consensus_peers.cc:597] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edb49884389c4dba957f0ee5af4ae0e4. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:26.868024 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e38ef2ac19d14f9c91fa8abe86cc7987" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:26.873914 1510 raft_consensus.cc:2468] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:26.874351 1586 leader_election.cc:304] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:26.874569 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1033) took 5881 ms. Trace:
I20251024 19:48:26.874678 1775 rpcz_store.cc:276] 1024 19:48:20.992947 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:20.993013 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:26.798685 (+5805672us) ts_tablet_manager.cc:1151] Deleting tablet 23d4dd0b42d343cb9354ac698a8c761f
1024 19:48:26.798698 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.827159 (+ 28461us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.836474 (+ 9315us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.874548 (+ 38074us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5805425}
I20251024 19:48:26.874841 3573 raft_consensus.cc:2804] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:26.874918 3573 raft_consensus.cc:493] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:26.874984 3573 raft_consensus.cc:3060] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.875167 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 23d4dd0b42d343cb9354ac698a8c761f (table default.Table_628b2afa9401426980be0eae5dffd917 [id=5b96ec1514ae41e3a0103e4b381aab61]) successfully deleted
I20251024 19:48:26.878856 3573 raft_consensus.cc:515] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.880867 3573 leader_election.cc:290] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:26.882005 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e38ef2ac19d14f9c91fa8abe86cc7987" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:26.882999 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e38ef2ac19d14f9c91fa8abe86cc7987" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:26.883214 1510 raft_consensus.cc:3060] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.886222 1775 tablet_replica.cc:333] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:26.886679 1775 raft_consensus.cc:2243] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.886904 1775 raft_consensus.cc:2272] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.887976 1775 ts_tablet_manager.cc:1916] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:26.888152 1586 leader_election.cc:343] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:26.891160 1510 raft_consensus.cc:2468] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:26.891784 1586 leader_election.cc:304] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:26.892067 3573 raft_consensus.cc:2804] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:26.892171 3573 raft_consensus.cc:697] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:26.892445 3573 consensus_queue.cc:237] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.905102 1775 ts_tablet_manager.cc:1929] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.905318 1775 log.cc:1199] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/d1d26e02ad2945b39cf27738ed5b7cf4
I20251024 19:48:26.905798 1775 ts_tablet_manager.cc:1950] T d1d26e02ad2945b39cf27738ed5b7cf4 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:26.906581 1776 ts_tablet_manager.cc:1929] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:26.908644 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1036) took 5865 ms. Trace:
I20251024 19:48:26.908735 1775 rpcz_store.cc:276] 1024 19:48:21.042960 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.043734 (+ 774us) service_pool.cc:225] Handling call
1024 19:48:26.881687 (+5837953us) ts_tablet_manager.cc:1151] Deleting tablet d1d26e02ad2945b39cf27738ed5b7cf4
1024 19:48:26.881701 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.891801 (+ 10100us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.904965 (+ 13164us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.908619 (+ 3654us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5837729}
I20251024 19:48:26.909214 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet d1d26e02ad2945b39cf27738ed5b7cf4 (table default.Table_2c02fd9cab9b479d8876a6dcbd586b17 [id=3660d29be985453fa1741ee293ac2a14]) successfully deleted
I20251024 19:48:26.906761 1776 log.cc:1199] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/96a6e0a4882c42399fd07f06ac21aa3f
I20251024 19:48:26.911310 1776 ts_tablet_manager.cc:1950] T 96a6e0a4882c42399fd07f06ac21aa3f P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:26.912690 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1191) took 1993 ms. Trace:
I20251024 19:48:26.912850 1776 rpcz_store.cc:276] 1024 19:48:24.919081 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.919192 (+ 111us) service_pool.cc:225] Handling call
1024 19:48:26.802314 (+1883122us) ts_tablet_manager.cc:1151] Deleting tablet 96a6e0a4882c42399fd07f06ac21aa3f
1024 19:48:26.802324 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.843901 (+ 41577us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.906441 (+ 62540us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.912669 (+ 6228us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1879270}
I20251024 19:48:26.916975 1768 tablet_bootstrap.cc:654] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.918146 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1040) took 5781 ms. Trace:
I20251024 19:48:26.923379 1775 rpcz_store.cc:276] 1024 19:48:21.137097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.137172 (+ 75us) service_pool.cc:225] Handling call
1024 19:48:26.917989 (+5780817us) ts_tablet_manager.cc:1151] Deleting tablet 7ecf3276437a43cf84a47a0e66553e0a
1024 19:48:26.918003 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.918121 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5780600}
W20251024 19:48:26.924047 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7ecf3276437a43cf84a47a0e66553e0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ecf3276437a43cf84a47a0e66553e0a already in progress: creating tablet
I20251024 19:48:26.924453 1775 tablet_replica.cc:333] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:26.924696 1775 raft_consensus.cc:2243] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:26.924904 1775 raft_consensus.cc:2272] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.924903 3538 consensus_queue.cc:1048] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:26.928520 1775 ts_tablet_manager.cc:1916] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.937939 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 96a6e0a4882c42399fd07f06ac21aa3f (table default.Table_19bbc037a7f241f5b338f4cfa4ebe82d [id=b2ce46e067994c9fa4df9f00de59d68f]) successfully deleted
I20251024 19:48:26.938560 1776 tablet_replica.cc:333] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:26.939144 1776 raft_consensus.cc:2243] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:26.939939 1776 raft_consensus.cc:2272] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.941951 1776 ts_tablet_manager.cc:1916] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.927490 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cbb82d8833d46d7ab2c87c26c99fcbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:26.946219 1775 ts_tablet_manager.cc:1929] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:26.953864 1775 log.cc:1199] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/45cff9bcc4924e0081518c284f074940
I20251024 19:48:26.953208 1766 tablet_bootstrap.cc:492] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:26.954169 1766 ts_tablet_manager.cc:1403] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.304s user 0.004s sys 0.000s
I20251024 19:48:26.955041 1766 raft_consensus.cc:359] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.955294 1766 raft_consensus.cc:385] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.955322 1767 tablet_bootstrap.cc:492] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:26.955461 1767 ts_tablet_manager.cc:1403] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.214s user 0.011s sys 0.011s
I20251024 19:48:26.955490 1766 raft_consensus.cc:740] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.955765 1766 consensus_queue.cc:260] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.956323 1766 ts_tablet_manager.cc:1434] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:26.956616 1766 tablet_bootstrap.cc:492] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:26.958269 1766 tablet_bootstrap.cc:654] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.958654 1775 ts_tablet_manager.cc:1950] T 45cff9bcc4924e0081518c284f074940 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:26.958702 1767 raft_consensus.cc:359] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.958892 1767 raft_consensus.cc:385] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:26.959379 1767 raft_consensus.cc:740] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:26.959612 1767 consensus_queue.cc:260] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.960084 1767 ts_tablet_manager.cc:1434] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20251024 19:48:26.960371 1767 tablet_bootstrap.cc:492] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:26.961838 1767 tablet_bootstrap.cc:654] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:26.968763 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1041) took 5831 ms. Trace:
I20251024 19:48:26.969630 1775 rpcz_store.cc:276] 1024 19:48:21.137210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.137643 (+ 433us) service_pool.cc:225] Handling call
1024 19:48:26.924399 (+5786756us) ts_tablet_manager.cc:1151] Deleting tablet 45cff9bcc4924e0081518c284f074940
1024 19:48:26.924410 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.935774 (+ 11364us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.946106 (+ 10332us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.968728 (+ 22622us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5786633}
I20251024 19:48:26.970170 1775 tablet_replica.cc:333] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:26.970129 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 45cff9bcc4924e0081518c284f074940 (table default.Table_34ef66bee6bd460b92cf30b469b3f179 [id=32d4cdae3aac408a88b0a207c8c08fcd]) successfully deleted
I20251024 19:48:26.970726 1775 raft_consensus.cc:2243] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:26.971478 1775 pending_rounds.cc:70] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e: Trying to abort 1 pending ops.
I20251024 19:48:26.972020 1775 pending_rounds.cc:77] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429413847891968 op_type: NO_OP noop_request { }
I20251024 19:48:26.972379 1775 raft_consensus.cc:2889] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:26.972685 1775 raft_consensus.cc:2272] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:26.973757 1775 ts_tablet_manager.cc:1916] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:26.976016 3528 raft_consensus.cc:493] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:26.976155 3528 raft_consensus.cc:515] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.976776 3528 leader_election.cc:290] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:26.977667 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee3c521876324808857f7724028c895f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:26.977869 1370 raft_consensus.cc:2468] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:26.978330 1586 leader_election.cc:304] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:26.978626 3528 raft_consensus.cc:2804] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:26.978696 3528 raft_consensus.cc:493] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:26.978756 3528 raft_consensus.cc:3060] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.981386 3528 raft_consensus.cc:515] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.982650 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee3c521876324808857f7724028c895f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:26.982820 1370 raft_consensus.cc:3060] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:26.985559 1370 raft_consensus.cc:2468] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:26.986136 1586 leader_election.cc:304] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:26.986397 3573 raft_consensus.cc:2804] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:26.986557 3573 raft_consensus.cc:697] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:26.986801 3573 consensus_queue.cc:237] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:26.988950 3528 leader_election.cc:290] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:26.989832 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 60fbfb1620934034af2acaa39677554f
W20251024 19:48:26.990823 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 73ab749795a64339abeaf1ce5d6570ab consensus state has no leader
I20251024 19:48:26.998343 1776 ts_tablet_manager.cc:1929] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.997541 1766 tablet_bootstrap.cc:492] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:26.999181 1775 ts_tablet_manager.cc:1929] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:26.999511 1775 log.cc:1199] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/30bce965315d43948ecb8282c829c557
I20251024 19:48:26.999539 1776 log.cc:1199] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/3fd8aa1d9aca429dbfe46f472ebbf53b
I20251024 19:48:26.999290 1766 ts_tablet_manager.cc:1403] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.005s
I20251024 19:48:27.000319 1775 ts_tablet_manager.cc:1950] T 30bce965315d43948ecb8282c829c557 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:27.000574 1766 raft_consensus.cc:359] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.000741 1766 raft_consensus.cc:385] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.000787 1766 raft_consensus.cc:740] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.001374 1776 ts_tablet_manager.cc:1950] T 3fd8aa1d9aca429dbfe46f472ebbf53b P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:27.002166 3538 raft_consensus.cc:493] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.002333 3538 raft_consensus.cc:515] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.002866 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1192) took 2083 ms. Trace:
I20251024 19:48:27.002918 3538 leader_election.cc:290] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.002952 1776 rpcz_store.cc:276] 1024 19:48:24.919737 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.919793 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:26.938489 (+2018696us) ts_tablet_manager.cc:1151] Deleting tablet 3fd8aa1d9aca429dbfe46f472ebbf53b
1024 19:48:26.938504 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.973223 (+ 34719us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.998239 (+ 25016us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.002844 (+ 4605us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2015290}
I20251024 19:48:27.003693 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0797d14e9fd846788164747523df0e2c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:27.004037 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0797d14e9fd846788164747523df0e2c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:27.004268 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 60fbfb1620934034af2acaa39677554f with delete_type TABLET_DATA_DELETED (replaced by tablet 1286dac5e8d24d50a652d3a485814159) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:27.004583 1444 leader_election.cc:343] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:27.005069 1447 leader_election.cc:343] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.005189 1447 leader_election.cc:304] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.004608 1766 consensus_queue.cc:260] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.005589 3456 raft_consensus.cc:2749] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:27.006409 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 60fbfb1620934034af2acaa39677554f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60fbfb1620934034af2acaa39677554f already in progress: creating tablet
I20251024 19:48:27.006732 1776 tablet_replica.cc:333] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:27.007819 1776 raft_consensus.cc:2243] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.008291 1766 ts_tablet_manager.cc:1434] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251024 19:48:27.008826 1776 raft_consensus.cc:2272] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:27.009344 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0cbe0a3436a498a84c892ac156798ed consensus state has no leader
I20251024 19:48:27.009768 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f6f3683609784447acb05ef28817aa01" } new_table_name: "default.Table_46d9253b20134c8b9329d03b90cca0ec" modify_external_catalogs: true
I20251024 19:48:27.010154 3528 raft_consensus.cc:493] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.010289 3528 raft_consensus.cc:515] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.011041 3528 leader_election.cc:290] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.011610 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc7ccf1495924782b9b6ac1173bc73a1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:27.011795 1370 raft_consensus.cc:2468] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:27.012303 1586 leader_election.cc:304] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:27.012574 3528 raft_consensus.cc:2804] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:27.012678 3528 raft_consensus.cc:493] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:27.012742 3528 raft_consensus.cc:3060] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:27.012941 1776 ts_tablet_manager.cc:1916] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.016515 3528 raft_consensus.cc:515] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.017447 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0ef705da3bc84def887ca8ae8ee4a878"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:27.017642 3528 leader_election.cc:290] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:27.018325 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 505ba3165385434faea556e97f42a7f6 consensus state has no leader
I20251024 19:48:27.018297 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc7ccf1495924782b9b6ac1173bc73a1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:27.018491 1370 raft_consensus.cc:3060] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:27.018838 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_16bdfd2e16724b5a8baf4bf776eaa165"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:27.019906 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_bedd8edc24014953ac8c7dcecf18f128" } new_table_name: "default.Table_45d12e1dd63849c783278f2435d7c3a6" modify_external_catalogs: true
I20251024 19:48:27.021338 1370 raft_consensus.cc:2468] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:27.021895 1586 leader_election.cc:304] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:27.022480 3528 raft_consensus.cc:2804] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:27.022686 3528 raft_consensus.cc:697] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:27.023191 3528 consensus_queue.cc:237] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.028285 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ad21fd1f5d1f45fb99a0560e3405d69f
I20251024 19:48:27.029191 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7065c04cd8e6495180d009455573cde0" } modify_external_catalogs: true
I20251024 19:48:27.029405 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7065c04cd8e6495180d009455573cde0" } modify_external_catalogs: true
W20251024 19:48:27.029847 1447 consensus_peers.cc:597] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:27.035238 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c3453ee396f645c79eb00b1777d54f12" } modify_external_catalogs: true
I20251024 19:48:27.035404 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_c3453ee396f645c79eb00b1777d54f12" } modify_external_catalogs: true
I20251024 19:48:27.040856 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_498bb6a3b91644e8831918efd611298d" } new_table_name: "default.Table_fbb60f8722b645e79fb36518d6cc696d" modify_external_catalogs: true
I20251024 19:48:27.044390 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet ad21fd1f5d1f45fb99a0560e3405d69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:27.045145 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet ad21fd1f5d1f45fb99a0560e3405d69f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad21fd1f5d1f45fb99a0560e3405d69f already in progress: creating tablet
I20251024 19:48:27.045953 1776 ts_tablet_manager.cc:1929] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:27.046393 1776 log.cc:1199] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/45cff9bcc4924e0081518c284f074940
I20251024 19:48:27.047664 1776 ts_tablet_manager.cc:1950] T 45cff9bcc4924e0081518c284f074940 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:27.048134 1185 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_d0930167e31244e789c97b3134c4b7a6 [id=9065c680f4da42b5a6e5ca335c8e2f3b] alter complete (version 4)
I20251024 19:48:27.048674 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2fb3f9af398745dcafa14f615ef6a116" } modify_external_catalogs: true
I20251024 19:48:27.048904 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2fb3f9af398745dcafa14f615ef6a116" } modify_external_catalogs: true
I20251024 19:48:27.052809 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1194) took 2132 ms. Trace:
I20251024 19:48:27.053226 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet bd4deeaab67442898911dabfa7302492
I20251024 19:48:27.053216 1776 rpcz_store.cc:276] 1024 19:48:24.920304 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.920486 (+ 182us) service_pool.cc:225] Handling call
1024 19:48:27.006335 (+2085849us) ts_tablet_manager.cc:1151] Deleting tablet 45cff9bcc4924e0081518c284f074940
1024 19:48:27.006348 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.032362 (+ 26014us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.045457 (+ 13095us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.052697 (+ 7240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2082935}
I20251024 19:48:27.054199 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 60f591124dbb444fa7a5b7f408d575df
I20251024 19:48:27.054917 1776 tablet_replica.cc:333] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:27.054993 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 60f591124dbb444fa7a5b7f408d575df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:27.055594 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 60f591124dbb444fa7a5b7f408d575df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60f591124dbb444fa7a5b7f408d575df already in progress: creating tablet
I20251024 19:48:27.055753 1776 raft_consensus.cc:2243] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:27.056917 1776 raft_consensus.cc:2272] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.059362 1776 ts_tablet_manager.cc:1916] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.064939 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet bd4deeaab67442898911dabfa7302492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:27.065801 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet bd4deeaab67442898911dabfa7302492 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4deeaab67442898911dabfa7302492 already in progress: creating tablet
I20251024 19:48:27.070991 3466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.252.212:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.252.254:36505,127.0.252.253:41375,127.0.252.252:36627
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.252.253:41375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.252.253
--webserver_port=43901
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 24 Oct 2025 19:43:25 UTC on 24a791456cd2
build id 8700
ASAN enabled
I20251024 19:48:27.074121 1768 tablet_bootstrap.cc:492] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.074430 1768 ts_tablet_manager.cc:1403] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.234s user 0.033s sys 0.036s
I20251024 19:48:27.074630 1188 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 2671108031db449cae2135835fc5f739 (table default.Table_d3ffdf5cbfd84551b1e314d4a772be1a [id=2f6b7b685af84337bd64141ef2a9e992]). Expected version 2 got 0
I20251024 19:48:27.074833 1188 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 6ef2add387da44868dbd2b5724ee7d86 (table default.Table_256d5a49b85c4a07b9c5e1839ee9bee6 [id=e55575a8212b40e093444ad77f9835e5]). Expected version 1 got 0
I20251024 19:48:27.075505 1768 raft_consensus.cc:359] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.075811 1768 raft_consensus.cc:385] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.075933 1768 raft_consensus.cc:740] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.076272 1768 consensus_queue.cc:260] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.076967 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet bc31e6c25ad14947af20abd766f12e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.077809 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d26e02ad2945b39cf27738ed5b7cf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.078102 1768 ts_tablet_manager.cc:1434] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:27.003376 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 3fd8aa1d9aca429dbfe46f472ebbf53b (table default.Table_003cb0c156394f9b8b0ba1b4ac52ce4c [id=b81a7d71911844c1be495ade2ceaf1d4]) successfully deleted
W20251024 19:48:27.078548 1176 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 75177.9us
I20251024 19:48:27.078931 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 45cff9bcc4924e0081518c284f074940 (table default.Table_34ef66bee6bd460b92cf30b469b3f179 [id=32d4cdae3aac408a88b0a207c8c08fcd]) successfully deleted
I20251024 19:48:27.079191 1768 tablet_bootstrap.cc:492] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:27.080108 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16bdfd2e16724b5a8baf4bf776eaa165 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.084532 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 60fbfb1620934034af2acaa39677554f with delete_type TABLET_DATA_DELETED (replaced by tablet 1286dac5e8d24d50a652d3a485814159) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.084635 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 60fbfb1620934034af2acaa39677554f with delete_type TABLET_DATA_DELETED (replaced by tablet 1286dac5e8d24d50a652d3a485814159) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.084805 1187 catalog_manager.cc:6719] ReplaceTablet: tablet 60fbfb1620934034af2acaa39677554f deleted and replaced by tablet 1286dac5e8d24d50a652d3a485814159
I20251024 19:48:27.084874 1768 tablet_bootstrap.cc:654] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.084060 1776 ts_tablet_manager.cc:1929] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:27.085101 1776 log.cc:1199] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/da7d38b63708427982591da1014aca81
I20251024 19:48:27.085655 1776 ts_tablet_manager.cc:1950] T da7d38b63708427982591da1014aca81 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:27.086882 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1196) took 2166 ms. Trace:
I20251024 19:48:27.087807 1776 rpcz_store.cc:276] 1024 19:48:24.920726 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.920777 (+ 51us) service_pool.cc:225] Handling call
1024 19:48:27.054760 (+2133983us) ts_tablet_manager.cc:1151] Deleting tablet da7d38b63708427982591da1014aca81
1024 19:48:27.054775 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.073284 (+ 18509us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.083952 (+ 10668us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.086864 (+ 2912us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2130988}
I20251024 19:48:27.088507 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet da7d38b63708427982591da1014aca81 (table default.Table_5744ec49c5e349d79d7473d6ed4154c6 [id=334320a439bf4611ba5b1aa660844023]) successfully deleted
I20251024 19:48:27.088697 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet 3e65583d0ea84a3ea10ae11462eac663 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:27.090194 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet ad21fd1f5d1f45fb99a0560e3405d69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.091038 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet d933103af272429c95f8d3fc54e427da from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:27.091077 1776 tablet_replica.cc:333] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:27.091488 1776 raft_consensus.cc:2243] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:27.092036 1776 raft_consensus.cc:2272] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.092933 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet ad21fd1f5d1f45fb99a0560e3405d69f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:26 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:27.093240 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ef705da3bc84def887ca8ae8ee4a878 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.093881 1776 ts_tablet_manager.cc:1916] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.097393 1768 tablet_bootstrap.cc:492] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.097545 1768 ts_tablet_manager.cc:1403] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.002s
I20251024 19:48:27.098323 1768 raft_consensus.cc:359] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.098495 1768 raft_consensus.cc:385] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.098558 1768 raft_consensus.cc:740] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.098805 1768 consensus_queue.cc:260] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.099648 1768 ts_tablet_manager.cc:1434] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:27.101383 1768 tablet_bootstrap.cc:492] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:27.101912 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet e38ef2ac19d14f9c91fa8abe86cc7987 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1a3523954448de93357510e6c1ec4e) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.102123 1774 tablet_replica.cc:333] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:27.102344 1774 raft_consensus.cc:2243] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.102510 1774 raft_consensus.cc:2272] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.103294 1768 tablet_bootstrap.cc:654] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.106926 1776 ts_tablet_manager.cc:1929] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:27.106940 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6c5115e46f6746acb0d4fa01a9227c6a" } modify_external_catalogs: true
I20251024 19:48:27.107203 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6c5115e46f6746acb0d4fa01a9227c6a" } modify_external_catalogs: true
I20251024 19:48:27.108692 1776 log.cc:1199] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/d1d26e02ad2945b39cf27738ed5b7cf4
I20251024 19:48:27.109241 1776 ts_tablet_manager.cc:1950] T d1d26e02ad2945b39cf27738ed5b7cf4 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:27.110136 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1197) took 2188 ms. Trace:
I20251024 19:48:27.110917 1776 rpcz_store.cc:276] 1024 19:48:24.922008 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.922083 (+ 75us) service_pool.cc:225] Handling call
1024 19:48:27.091031 (+2168948us) ts_tablet_manager.cc:1151] Deleting tablet d1d26e02ad2945b39cf27738ed5b7cf4
1024 19:48:27.091040 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.100649 (+ 9609us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.106821 (+ 6172us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.110116 (+ 3295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2167236}
I20251024 19:48:27.111547 3466 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 19:48:27.111754 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet d1d26e02ad2945b39cf27738ed5b7cf4 (table default.Table_2c02fd9cab9b479d8876a6dcbd586b17 [id=3660d29be985453fa1741ee293ac2a14]) successfully deleted
I20251024 19:48:27.111910 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet d933103af272429c95f8d3fc54e427da
I20251024 19:48:27.113085 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet d933103af272429c95f8d3fc54e427da with delete_type TABLET_DATA_DELETED (replaced by tablet eb9801bfcb5c42878ecf1b4d3605fce0) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.113147 1768 tablet_bootstrap.cc:492] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.113198 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet d933103af272429c95f8d3fc54e427da with delete_type TABLET_DATA_DELETED (replaced by tablet eb9801bfcb5c42878ecf1b4d3605fce0) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.113273 1768 ts_tablet_manager.cc:1403] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20251024 19:48:27.113462 1190 catalog_manager.cc:6719] ReplaceTablet: tablet d933103af272429c95f8d3fc54e427da deleted and replaced by tablet eb9801bfcb5c42878ecf1b4d3605fce0
I20251024 19:48:27.114234 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a67503cccddd4a6e9d8ea3b30488cc14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:27.114943 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a67503cccddd4a6e9d8ea3b30488cc14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.115183 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet d933103af272429c95f8d3fc54e427da with delete_type TABLET_DATA_DELETED (replaced by tablet eb9801bfcb5c42878ecf1b4d3605fce0) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.115132 1768 raft_consensus.cc:359] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.115365 1768 raft_consensus.cc:385] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.115430 1768 raft_consensus.cc:740] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.115607 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet fc7ccf1495924782b9b6ac1173bc73a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.115654 1768 consensus_queue.cc:260] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.116362 1768 ts_tablet_manager.cc:1434] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:27.116334 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 60f591124dbb444fa7a5b7f408d575df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.116706 1768 tablet_bootstrap.cc:492] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:27.116670 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 60f591124dbb444fa7a5b7f408d575df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.118103 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet bd4deeaab67442898911dabfa7302492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.118537 1768 tablet_bootstrap.cc:654] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.118561 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet bd4deeaab67442898911dabfa7302492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.119256 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet ee3c521876324808857f7724028c895f with delete_type TABLET_DATA_DELETED (replaced by tablet 94b35252e63f4a8ba305cc029ddb5ae8) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.126749 1774 ts_tablet_manager.cc:1916] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.130736 3466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251024 19:48:27.134815 1774 ts_tablet_manager.cc:1929] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:27.135020 1774 log.cc:1199] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/e38ef2ac19d14f9c91fa8abe86cc7987
I20251024 19:48:27.135596 1774 ts_tablet_manager.cc:1950] T e38ef2ac19d14f9c91fa8abe86cc7987 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:27.136675 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1198) took 2214 ms. Trace:
I20251024 19:48:27.136849 1776 rpcz_store.cc:276] 1024 19:48:24.922143 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.922185 (+ 42us) service_pool.cc:225] Handling call
1024 19:48:27.136522 (+2214337us) ts_tablet_manager.cc:1151] Deleting tablet 1edc55caab904722bf19a1c71df601cf
1024 19:48:27.136539 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.136647 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2212623}
I20251024 19:48:27.137420 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1199) took 2215 ms. Trace:
I20251024 19:48:27.137534 1776 rpcz_store.cc:276] 1024 19:48:24.922255 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.922296 (+ 41us) service_pool.cc:225] Handling call
1024 19:48:27.137309 (+2215013us) ts_tablet_manager.cc:1151] Deleting tablet d840888183ec4ef7bb245363c5bac322
1024 19:48:27.137321 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.137391 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2213321}
I20251024 19:48:27.137933 1776 tablet_replica.cc:333] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:27.138197 1776 raft_consensus.cc:2243] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.138420 1776 raft_consensus.cc:2272] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.138792 1774 tablet_replica.cc:333] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:27.139130 1774 raft_consensus.cc:2243] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.139282 1776 ts_tablet_manager.cc:1916] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.139416 1774 raft_consensus.cc:2272] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.145522 1774 ts_tablet_manager.cc:1916] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.148489 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee3c521876324808857f7724028c895f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:27.148782 1510 raft_consensus.cc:2468] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:27.149106 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee3c521876324808857f7724028c895f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:27.149276 1508 raft_consensus.cc:3060] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:27.152693 1511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc7ccf1495924782b9b6ac1173bc73a1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:27.152922 1511 raft_consensus.cc:2468] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:27.153119 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc7ccf1495924782b9b6ac1173bc73a1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:27.153252 1507 raft_consensus.cc:3060] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:27.156281 1507 raft_consensus.cc:2468] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:27.157380 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1042) took 5788 ms. Trace:
I20251024 19:48:27.157492 1775 rpcz_store.cc:276] 1024 19:48:21.369093 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.369146 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:26.970115 (+5600969us) ts_tablet_manager.cc:1151] Deleting tablet 30bce965315d43948ecb8282c829c557
1024 19:48:26.970126 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:26.980407 (+ 10281us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:26.994458 (+ 14051us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.157360 (+162902us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5600728}
I20251024 19:48:27.157917 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1043) took 5769 ms. Trace:
I20251024 19:48:27.157990 1775 rpcz_store.cc:276] 1024 19:48:21.388273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.390953 (+ 2680us) service_pool.cc:225] Handling call
1024 19:48:27.157798 (+5766845us) ts_tablet_manager.cc:1151] Deleting tablet 672b571081ea4cd78028b6a25625103c
1024 19:48:27.157807 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.157901 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5766573}
I20251024 19:48:27.158310 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1044) took 5769 ms. Trace:
I20251024 19:48:27.158373 1775 rpcz_store.cc:276] 1024 19:48:21.388804 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.391268 (+ 2464us) service_pool.cc:225] Handling call
1024 19:48:27.158215 (+5766947us) ts_tablet_manager.cc:1151] Deleting tablet 04ae62c9bf6f44108d57d0760283d063
1024 19:48:27.158223 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.158296 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5766738}
I20251024 19:48:27.158674 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1045) took 5765 ms. Trace:
I20251024 19:48:27.158737 1775 rpcz_store.cc:276] 1024 19:48:21.393653 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.393727 (+ 74us) service_pool.cc:225] Handling call
1024 19:48:27.158576 (+5764849us) ts_tablet_manager.cc:1151] Deleting tablet e0b57403cdf947b390a44a9889f2a2c3
1024 19:48:27.158584 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.158660 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5764637}
I20251024 19:48:27.159014 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1046) took 5763 ms. Trace:
I20251024 19:48:27.159071 1775 rpcz_store.cc:276] 1024 19:48:21.395835 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.395905 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:27.158919 (+5763014us) ts_tablet_manager.cc:1151] Deleting tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253
1024 19:48:27.158927 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.159002 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5762833}
I20251024 19:48:27.159364 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1047) took 5763 ms. Trace:
I20251024 19:48:27.159446 1775 rpcz_store.cc:276] 1024 19:48:21.395968 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.396117 (+ 149us) service_pool.cc:225] Handling call
1024 19:48:27.159253 (+5763136us) ts_tablet_manager.cc:1151] Deleting tablet 59eca6bb1dd84c67a7e53131abaa937a
1024 19:48:27.159261 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.159348 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5762950}
I20251024 19:48:27.159715 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1048) took 5763 ms. Trace:
I20251024 19:48:27.159776 1775 rpcz_store.cc:276] 1024 19:48:21.396065 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.396374 (+ 309us) service_pool.cc:225] Handling call
1024 19:48:27.159627 (+5763253us) ts_tablet_manager.cc:1151] Deleting tablet 3a99e9b3fe044c7eb14fa9bffa903873
1024 19:48:27.159636 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.159702 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5763122}
I20251024 19:48:27.160193 1766 tablet_bootstrap.cc:492] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:27.161799 1508 raft_consensus.cc:2468] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:27.163379 1766 tablet_bootstrap.cc:654] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.159991 1775 tablet_replica.cc:333] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:27.168428 1775 raft_consensus.cc:2243] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.168711 1775 raft_consensus.cc:2272] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.169386 1775 ts_tablet_manager.cc:1916] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.137722 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet e38ef2ac19d14f9c91fa8abe86cc7987 (table default.Table_6bb9ce85a6e341e7b2cc6d9ffc2c0127 [id=50a4f64cc7504ea190832dd265c81caf]) successfully deleted
I20251024 19:48:27.174988 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 30bce965315d43948ecb8282c829c557 (table default.Table_21c23f21b4f24c9989cdfe18a12a4693 [id=d9ae61ad6f8545d8a9945f08bdd731a3]) successfully deleted
W20251024 19:48:27.175187 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 672b571081ea4cd78028b6a25625103c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 672b571081ea4cd78028b6a25625103c already in progress: creating tablet
W20251024 19:48:27.175433 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 04ae62c9bf6f44108d57d0760283d063 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04ae62c9bf6f44108d57d0760283d063 already in progress: creating tablet
W20251024 19:48:27.175597 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet e0b57403cdf947b390a44a9889f2a2c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0b57403cdf947b390a44a9889f2a2c3 already in progress: creating tablet
W20251024 19:48:27.175750 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d9eab9aee9d47abbc3a6ae5d4e9f253 already in progress: creating tablet
W20251024 19:48:27.175979 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 59eca6bb1dd84c67a7e53131abaa937a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59eca6bb1dd84c67a7e53131abaa937a already in progress: creating tablet
W20251024 19:48:27.176211 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3a99e9b3fe044c7eb14fa9bffa903873 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a99e9b3fe044c7eb14fa9bffa903873 already in progress: creating tablet
W20251024 19:48:27.176446 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet d933103af272429c95f8d3fc54e427da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d933103af272429c95f8d3fc54e427da already in progress: creating tablet
I20251024 19:48:27.117439 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2d40dad65a34427ca08b38d576c386aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:27.179390 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d40dad65a34427ca08b38d576c386aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.189291 1774 ts_tablet_manager.cc:1929] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:27.191414 1775 ts_tablet_manager.cc:1929] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:27.191749 1775 log.cc:1199] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/da7d38b63708427982591da1014aca81
I20251024 19:48:27.191213 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_09fd5426c34f40bc9ae783f13695e8bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:27.192761 1775 ts_tablet_manager.cc:1950] T da7d38b63708427982591da1014aca81 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
W20251024 19:48:27.196156 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09fd5426c34f40bc9ae783f13695e8bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.195227 1766 tablet_bootstrap.cc:492] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:27.196486 1766 ts_tablet_manager.cc:1403] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.008s
I20251024 19:48:27.197551 1766 raft_consensus.cc:359] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:27.197866 1766 raft_consensus.cc:385] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.197983 1766 raft_consensus.cc:740] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.198350 1766 consensus_queue.cc:260] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:27.201998 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e3a8d735a1741cabcabd64964f172fd consensus state has no leader
I20251024 19:48:27.202932 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 1e45eefeeff949139f9407f9f90fba2a
I20251024 19:48:27.203555 1766 ts_tablet_manager.cc:1434] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251024 19:48:27.205093 1766 tablet_bootstrap.cc:492] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:27.206025 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_97c3d690de704e908a7ca2e664236bd1" } new_table_name: "default.Table_1c0473fd8e854921a389643b531f29a5" modify_external_catalogs: true
I20251024 19:48:27.207233 1766 tablet_bootstrap.cc:654] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.191239 1768 tablet_bootstrap.cc:492] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.208290 1774 log.cc:1199] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/fc7ccf1495924782b9b6ac1173bc73a1
I20251024 19:48:27.211453 3538 raft_consensus.cc:493] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.211603 3538 raft_consensus.cc:515] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.212270 3538 leader_election.cc:290] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.212917 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d1b6eaa353c4284a2b114b42437272c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:27.213716 1444 leader_election.cc:343] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.208360 1768 ts_tablet_manager.cc:1403] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.092s user 0.013s sys 0.035s
I20251024 19:48:27.204272 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e45eefeeff949139f9407f9f90fba2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.215159 1776 ts_tablet_manager.cc:1929] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:27.215263 1776 log.cc:1199] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/2e7b3288182744969e217d4f174fe4ea
I20251024 19:48:27.215582 1776 ts_tablet_manager.cc:1950] T 2e7b3288182744969e217d4f174fe4ea P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:27.213975 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d1b6eaa353c4284a2b114b42437272c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:27.214918 1768 raft_consensus.cc:359] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:27.216153 1768 raft_consensus.cc:385] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.216219 1768 raft_consensus.cc:740] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.216385 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1193) took 2296 ms. Trace:
I20251024 19:48:27.216396 1768 consensus_queue.cc:260] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:27.216464 1776 rpcz_store.cc:276] 1024 19:48:24.919988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.920061 (+ 73us) service_pool.cc:225] Handling call
1024 19:48:27.137877 (+2217816us) ts_tablet_manager.cc:1151] Deleting tablet 2e7b3288182744969e217d4f174fe4ea
1024 19:48:27.137888 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.143438 (+ 5550us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.215049 (+ 71611us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.216368 (+ 1319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2213801}
W20251024 19:48:27.216593 1447 leader_election.cc:343] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.216725 1447 leader_election.cc:304] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.216907 1768 ts_tablet_manager.cc:1434] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251024 19:48:27.217186 1768 tablet_bootstrap.cc:492] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:27.217442 3538 raft_consensus.cc:2749] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:27.218123 1774 ts_tablet_manager.cc:1950] T fc7ccf1495924782b9b6ac1173bc73a1 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:27.196206 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1053) took 5614 ms. Trace:
I20251024 19:48:27.218400 1775 rpcz_store.cc:276] 1024 19:48:21.581262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.581344 (+ 82us) service_pool.cc:225] Handling call
1024 19:48:27.159952 (+5578608us) ts_tablet_manager.cc:1151] Deleting tablet da7d38b63708427982591da1014aca81
1024 19:48:27.159960 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.178190 (+ 18230us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.190760 (+ 12570us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.196181 (+ 5421us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5578259}
I20251024 19:48:27.220309 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet fc7ccf1495924782b9b6ac1173bc73a1 (table default.Table_6a5150d7236d4d558c9f9fb4e70a78dd [id=991b0e462eba40ec882e422e60b43f26]) successfully deleted
I20251024 19:48:27.220834 1768 tablet_bootstrap.cc:654] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.221092 1774 tablet_replica.cc:333] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:27.221757 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet da7d38b63708427982591da1014aca81 (table default.Table_5744ec49c5e349d79d7473d6ed4154c6 [id=334320a439bf4611ba5b1aa660844023]) successfully deleted
I20251024 19:48:27.222098 1776 tablet_replica.cc:333] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:27.222388 1776 raft_consensus.cc:2243] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:27.222954 1776 raft_consensus.cc:2272] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.223692 1776 ts_tablet_manager.cc:1916] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.225395 1774 raft_consensus.cc:2243] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.227041 1774 raft_consensus.cc:2272] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.229684 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e45eefeeff949139f9407f9f90fba2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.230047 1774 ts_tablet_manager.cc:1916] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.230458 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet 19acf59898c8436e9d9fef98a7f07532 from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:27.236157 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 1edc55caab904722bf19a1c71df601cf because the tablet was not found. No further retry: Not found: Tablet not found: 1edc55caab904722bf19a1c71df601cf
W20251024 19:48:27.236403 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d840888183ec4ef7bb245363c5bac322 because the tablet was not found. No further retry: Not found: Tablet not found: d840888183ec4ef7bb245363c5bac322
I20251024 19:48:27.236565 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 2e7b3288182744969e217d4f174fe4ea (table default.Table_c4882268578941ccaf2c28be3074fdd2 [id=0b00aa7d2eaa4b1a967952b9a484fd42]) successfully deleted
I20251024 19:48:27.237690 1766 tablet_bootstrap.cc:492] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:27.237725 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e45eefeeff949139f9407f9f90fba2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.246026 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1051) took 5640 ms. Trace:
I20251024 19:48:27.246244 1775 rpcz_store.cc:276] 1024 19:48:21.580512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.582548 (+ 2036us) service_pool.cc:225] Handling call
1024 19:48:27.220740 (+5638192us) ts_tablet_manager.cc:1151] Deleting tablet 703441e1af8e4356afd22ef68dac87a2
1024 19:48:27.220752 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.220940 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5638023}
W20251024 19:48:27.246791 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 703441e1af8e4356afd22ef68dac87a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 703441e1af8e4356afd22ef68dac87a2 already in progress: creating tablet
I20251024 19:48:27.247184 1775 tablet_replica.cc:333] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:27.247433 1775 raft_consensus.cc:2243] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.247676 1775 raft_consensus.cc:2272] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.247913 1766 ts_tablet_manager.cc:1403] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.003s
I20251024 19:48:27.248800 1766 raft_consensus.cc:359] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.249146 1766 raft_consensus.cc:385] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.249260 1766 raft_consensus.cc:740] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.249943 1775 ts_tablet_manager.cc:1916] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.172804 1184 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 58a8f4f90f794e83a158e0026e5cea7f (table default.Table_c7b7dc69f2e640f6bb21208e8d8f32b8 [id=3fe81a9c4a134e908f4f2599c4b0cbf4]). Expected version 1 got 0
I20251024 19:48:27.253650 3530 raft_consensus.cc:493] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.253826 1766 consensus_queue.cc:260] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.253891 3530 raft_consensus.cc:515] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:27.254376 1766 ts_tablet_manager.cc:1434] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251024 19:48:27.254628 3530 leader_election.cc:290] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:27.174517 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet e320bd0c6eba4d1b965e31d16234abcf
I20251024 19:48:27.256558 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet e320bd0c6eba4d1b965e31d16234abcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.257387 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "769f576810ab49368eb0fc7837b7da58" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:27.258033 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet e320bd0c6eba4d1b965e31d16234abcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.258539 1766 tablet_bootstrap.cc:492] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:27.259436 1643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "769f576810ab49368eb0fc7837b7da58" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:27.260082 1305 leader_election.cc:304] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.260421 3530 raft_consensus.cc:2749] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:27.264292 1768 tablet_bootstrap.cc:492] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.265146 1768 ts_tablet_manager.cc:1403] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.010s
I20251024 19:48:27.266180 1768 raft_consensus.cc:359] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.267416 1768 raft_consensus.cc:385] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.267014 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 80c58e2439dc490c824b5f0023b5434b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.267536 3530 raft_consensus.cc:493] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.266367 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet e320bd0c6eba4d1b965e31d16234abcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.267767 1768 raft_consensus.cc:740] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.267943 3530 raft_consensus.cc:515] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.268131 1768 consensus_queue.cc:260] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.268548 3530 leader_election.cc:290] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:27.269954 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80c58e2439dc490c824b5f0023b5434b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:27.270470 1305 leader_election.cc:343] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:27.274178 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b224c639bb546aea1fdfd9a0bfc415b consensus state has no leader
I20251024 19:48:27.277719 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3e65583d0ea84a3ea10ae11462eac663
I20251024 19:48:27.278481 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e65583d0ea84a3ea10ae11462eac663 with delete_type TABLET_DATA_DELETED (replaced by tablet b9ff74454a7e4803a7d0c307a21aef7a) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.279124 1768 ts_tablet_manager.cc:1434] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20251024 19:48:27.279431 1768 tablet_bootstrap.cc:492] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:27.279604 1187 catalog_manager.cc:6719] ReplaceTablet: tablet 3e65583d0ea84a3ea10ae11462eac663 deleted and replaced by tablet b9ff74454a7e4803a7d0c307a21aef7a
I20251024 19:48:27.279775 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e65583d0ea84a3ea10ae11462eac663 with delete_type TABLET_DATA_DELETED (replaced by tablet b9ff74454a7e4803a7d0c307a21aef7a) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.280177 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9cbca7d5292d44c49d2d550aa9e18e92" } new_table_name: "default.Table_3aff5d7b2c404b0693d95663d21758ca" modify_external_catalogs: true
I20251024 19:48:27.284018 1768 tablet_bootstrap.cc:654] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.265553 1775 ts_tablet_manager.cc:1929] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:27.284472 1775 log.cc:1199] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/3fd8aa1d9aca429dbfe46f472ebbf53b
I20251024 19:48:27.285002 1775 ts_tablet_manager.cc:1950] T 3fd8aa1d9aca429dbfe46f472ebbf53b P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:27.286263 1774 ts_tablet_manager.cc:1929] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:27.267840 1766 tablet_bootstrap.cc:654] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.280582 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e65583d0ea84a3ea10ae11462eac663 with delete_type TABLET_DATA_DELETED (replaced by tablet b9ff74454a7e4803a7d0c307a21aef7a) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.269445 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80c58e2439dc490c824b5f0023b5434b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:27.290555 1508 raft_consensus.cc:2468] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:27.290746 1776 ts_tablet_manager.cc:1929] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:27.290863 1776 log.cc:1199] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/38818c7997c84cedad5f9b09eef77105
I20251024 19:48:27.291206 1776 ts_tablet_manager.cc:1950] T 38818c7997c84cedad5f9b09eef77105 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:27.291985 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1190) took 2375 ms. Trace:
I20251024 19:48:27.292074 1776 rpcz_store.cc:276] 1024 19:48:24.916660 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.919277 (+ 2617us) service_pool.cc:225] Handling call
1024 19:48:27.222044 (+2302767us) ts_tablet_manager.cc:1151] Deleting tablet 38818c7997c84cedad5f9b09eef77105
1024 19:48:27.222057 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.261437 (+ 39380us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.290632 (+ 29195us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.291966 (+ 1334us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2297883}
I20251024 19:48:27.292269 3404 raft_consensus.cc:493] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.292408 3404 raft_consensus.cc:515] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.292636 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 38818c7997c84cedad5f9b09eef77105 (table default.Table_d7a9ef5436984a1b83dbeac77c302549 [id=4c4868b6a38046faaf24f65fd00420cb]) successfully deleted
I20251024 19:48:27.292594 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_42d61eaa29c64c7281f5cb3cf361b9e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:27.293023 3404 leader_election.cc:290] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.293457 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1052) took 5712 ms. Trace:
W20251024 19:48:27.293632 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42d61eaa29c64c7281f5cb3cf361b9e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.291555 1308 leader_election.cc:304] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.293866 1774 log.cc:1199] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/ee3c521876324808857f7724028c895f
I20251024 19:48:27.294252 1774 ts_tablet_manager.cc:1950] T ee3c521876324808857f7724028c895f P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:27.293680 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1cdd5ec32ca49c1bfafaf3a24050e89" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:27.294680 1508 raft_consensus.cc:2468] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:27.295130 1775 rpcz_store.cc:276] 1024 19:48:21.580948 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.582743 (+ 1795us) service_pool.cc:225] Handling call
1024 19:48:27.247116 (+5664373us) ts_tablet_manager.cc:1151] Deleting tablet 3fd8aa1d9aca429dbfe46f472ebbf53b
1024 19:48:27.247128 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.253152 (+ 6024us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.265373 (+ 12221us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.293376 (+ 28003us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5664223}
I20251024 19:48:27.297930 1586 leader_election.cc:304] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:27.298226 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1cdd5ec32ca49c1bfafaf3a24050e89" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:27.298581 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet ee3c521876324808857f7724028c895f (table default.Table_a86db858d09749a9a4706c761c20ee9a [id=f49bfd0c428d44f69a3d97a6c51a380a]) successfully deleted
I20251024 19:48:27.298700 3530 raft_consensus.cc:2804] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:27.298771 3530 raft_consensus.cc:493] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:27.298832 3530 raft_consensus.cc:3060] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:27.299295 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 1e45eefeeff949139f9407f9f90fba2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e45eefeeff949139f9407f9f90fba2a already in progress: creating tablet
I20251024 19:48:27.299698 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 3fd8aa1d9aca429dbfe46f472ebbf53b (table default.Table_003cb0c156394f9b8b0ba1b4ac52ce4c [id=b81a7d71911844c1be495ade2ceaf1d4]) successfully deleted
I20251024 19:48:27.300062 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1060) took 5634 ms. Trace:
I20251024 19:48:27.300153 1775 rpcz_store.cc:276] 1024 19:48:21.665927 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.666196 (+ 269us) service_pool.cc:225] Handling call
1024 19:48:27.299920 (+5633724us) ts_tablet_manager.cc:1151] Deleting tablet 68d85226ac5b4e4ab22a49dc419e375a
1024 19:48:27.299931 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.300026 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5633580}
W20251024 19:48:27.300551 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 68d85226ac5b4e4ab22a49dc419e375a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68d85226ac5b4e4ab22a49dc419e375a already in progress: creating tablet
W20251024 19:48:27.301085 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet e320bd0c6eba4d1b965e31d16234abcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e320bd0c6eba4d1b965e31d16234abcf already in progress: creating tablet
I20251024 19:48:27.301374 1774 tablet_replica.cc:333] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:27.301546 3530 raft_consensus.cc:515] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.301882 1775 tablet_replica.cc:333] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:27.302162 3530 leader_election.cc:290] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:27.302603 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80c58e2439dc490c824b5f0023b5434b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
W20251024 19:48:27.303086 1305 leader_election.cc:343] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.303278 1774 raft_consensus.cc:2243] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.303936 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80c58e2439dc490c824b5f0023b5434b" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:27.304121 1508 raft_consensus.cc:3060] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:27.304651 1586 leader_election.cc:343] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.304981 3404 raft_consensus.cc:2804] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:27.305044 3404 raft_consensus.cc:493] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:27.305096 3404 raft_consensus.cc:3060] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:27.306200 3603 tablet.cc:1722] T 4ffd446c269448d38cc8d987a53aecf9 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:27.306919 1508 raft_consensus.cc:2468] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:27.307263 1775 raft_consensus.cc:2243] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:27.307740 1775 raft_consensus.cc:2272] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.308154 1308 leader_election.cc:304] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.308326 3404 raft_consensus.cc:515] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.309473 1774 raft_consensus.cc:2272] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.308902 1766 tablet_bootstrap.cc:492] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:27.310015 1766 ts_tablet_manager.cc:1403] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.002s
I20251024 19:48:27.309218 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1cdd5ec32ca49c1bfafaf3a24050e89" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:27.310153 1776 tablet_replica.cc:333] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:27.310205 1508 raft_consensus.cc:3060] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:27.310323 1776 raft_consensus.cc:2243] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.310468 1776 raft_consensus.cc:2272] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.310986 1776 ts_tablet_manager.cc:1916] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.313020 1508 raft_consensus.cc:2468] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:27.313429 1586 leader_election.cc:304] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:27.292057 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 769f576810ab49368eb0fc7837b7da58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.313731 3573 raft_consensus.cc:2804] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:27.313853 3573 raft_consensus.cc:697] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:27.314081 3573 consensus_queue.cc:237] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.314205 3404 leader_election.cc:290] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.315804 3456 consensus_queue.cc:1048] T ccd46da86fe444baa675d3391cdd410c P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:27.318224 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1cdd5ec32ca49c1bfafaf3a24050e89" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:27.323382 1766 raft_consensus.cc:359] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.324004 1766 raft_consensus.cc:385] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.324281 1766 raft_consensus.cc:740] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.324141 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d078179b97b04326aefe7ca93fdc335b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:27.324831 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d078179b97b04326aefe7ca93fdc335b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.324816 1766 consensus_queue.cc:260] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:27.326172 1586 leader_election.cc:343] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.326751 1766 ts_tablet_manager.cc:1434] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251024 19:48:27.327538 1766 tablet_bootstrap.cc:492] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:27.330679 3530 raft_consensus.cc:493] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.330907 3530 raft_consensus.cc:515] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.331595 3530 leader_election.cc:290] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:27.331997 1774 ts_tablet_manager.cc:1916] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.333117 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccd46da86fe444baa675d3391cdd410c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:27.323625 1775 ts_tablet_manager.cc:1916] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:27.336470 1447 consensus_peers.cc:597] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5cbce9e4b61451ba7f458df922be759. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:27.337342 1766 tablet_bootstrap.cc:654] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.333007 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3dbd05fe93094e11baed3b6447a56980" } new_table_name: "default.Table_707ea75759974f808c67129a8651b7b5" modify_external_catalogs: true
I20251024 19:48:27.347226 1768 tablet_bootstrap.cc:492] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.347405 1768 ts_tablet_manager.cc:1403] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.007s
I20251024 19:48:27.348150 1768 raft_consensus.cc:359] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.348347 1768 raft_consensus.cc:385] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.348439 1768 raft_consensus.cc:740] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.348655 1768 consensus_queue.cc:260] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.349361 1768 ts_tablet_manager.cc:1434] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:27.349771 1768 tablet_bootstrap.cc:492] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:27.336650 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_db1ad1d0cb0f4bfd9cb4fc16270c74d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:27.350999 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db1ad1d0cb0f4bfd9cb4fc16270c74d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.351680 1768 tablet_bootstrap.cc:654] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.354212 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet fc7ccf1495924782b9b6ac1173bc73a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.354491 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet d1d26e02ad2945b39cf27738ed5b7cf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.354598 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet ee3c521876324808857f7724028c895f with delete_type TABLET_DATA_DELETED (replaced by tablet 94b35252e63f4a8ba305cc029ddb5ae8) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.354806 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 38818c7997c84cedad5f9b09eef77105 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.354975 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet e38ef2ac19d14f9c91fa8abe86cc7987 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b1a3523954448de93357510e6c1ec4e) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.356208 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_312699096d58411cbc14672a7c9c7d9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:27.356137 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_07368d3d5ebf4c10baed00c0c48d9769"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:27.356775 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07368d3d5ebf4c10baed00c0c48d9769 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.360131 1776 ts_tablet_manager.cc:1929] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:27.360275 1776 log.cc:1199] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/23d4dd0b42d343cb9354ac698a8c761f
I20251024 19:48:27.360715 1776 ts_tablet_manager.cc:1950] T 23d4dd0b42d343cb9354ac698a8c761f P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:27.361613 1767 tablet_bootstrap.cc:492] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:27.361763 1767 ts_tablet_manager.cc:1403] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.401s user 0.006s sys 0.001s
I20251024 19:48:27.361799 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1204) took 2421 ms. Trace:
I20251024 19:48:27.361903 1776 rpcz_store.cc:276] 1024 19:48:24.940266 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.940323 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:27.310097 (+2369774us) ts_tablet_manager.cc:1151] Deleting tablet 23d4dd0b42d343cb9354ac698a8c761f
1024 19:48:27.310109 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.342730 (+ 32621us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.359980 (+ 17250us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.361771 (+ 1791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2369599}
I20251024 19:48:27.362499 1767 raft_consensus.cc:359] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.362669 1767 raft_consensus.cc:385] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.362732 1767 raft_consensus.cc:740] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.362962 1767 consensus_queue.cc:260] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.363642 1767 ts_tablet_manager.cc:1434] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:27.340260 3456 raft_consensus.cc:493] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.373688 3456 raft_consensus.cc:515] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.374367 3456 leader_election.cc:290] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.341032 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccd46da86fe444baa675d3391cdd410c" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:27.375643 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ef2add387da44868dbd2b5724ee7d86" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:27.376219 1447 leader_election.cc:343] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.378535 1308 leader_election.cc:304] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.379426 3599 raft_consensus.cc:2749] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:27.356786 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_312699096d58411cbc14672a7c9c7d9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.362447 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 23d4dd0b42d343cb9354ac698a8c761f (table default.Table_628b2afa9401426980be0eae5dffd917 [id=5b96ec1514ae41e3a0103e4b381aab61]) successfully deleted
I20251024 19:48:27.381529 1766 tablet_bootstrap.cc:492] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:27.381716 1766 ts_tablet_manager.cc:1403] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.002s
I20251024 19:48:27.382197 3601 tablet.cc:1722] T 4ffd446c269448d38cc8d987a53aecf9 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:27.382552 1766 raft_consensus.cc:359] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.382745 1766 raft_consensus.cc:385] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.382826 1766 raft_consensus.cc:740] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.385152 1766 consensus_queue.cc:260] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.385836 3573 raft_consensus.cc:493] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.385984 3573 raft_consensus.cc:515] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.386588 3573 leader_election.cc:290] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.387636 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f522a825264ea7b215304dbb7aa2d5" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:27.387816 1508 raft_consensus.cc:2468] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:27.387876 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f522a825264ea7b215304dbb7aa2d5" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:27.387905 3610 tablet.cc:1722] T 4ffd446c269448d38cc8d987a53aecf9 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:27.388242 1586 leader_election.cc:304] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
W20251024 19:48:27.388698 1586 leader_election.cc:343] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.389000 3573 raft_consensus.cc:2804] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:27.389066 3573 raft_consensus.cc:493] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:27.389120 3573 raft_consensus.cc:3060] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:27.392192 3573 raft_consensus.cc:515] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.393760 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f522a825264ea7b215304dbb7aa2d5" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:27.393970 1508 raft_consensus.cc:3060] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:27.393950 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f522a825264ea7b215304dbb7aa2d5" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:27.394445 1586 leader_election.cc:343] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.394591 3573 leader_election.cc:290] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:27.394979 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1dadb9331a5a45049c728783aa70b46d consensus state has no leader
I20251024 19:48:27.396188 1766 ts_tablet_manager.cc:1434] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251024 19:48:27.397078 1508 raft_consensus.cc:2468] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:27.397428 1586 leader_election.cc:304] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:27.397890 3573 raft_consensus.cc:2804] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
W20251024 19:48:27.398064 3615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 19:48:27.398339 3573 raft_consensus.cc:697] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:27.399166 3573 consensus_queue.cc:237] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:27.399820 3612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 19:48:27.400139 3466 server_base.cc:1047] running on GCE node
W20251024 19:48:27.401822 3611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 19:48:27.402748 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_06ba8e65390c454fb0c4168ec0872485" } new_table_name: "default.Table_99ca9081a08f46faaaa0ad678def273a" modify_external_catalogs: true
I20251024 19:48:27.404713 3466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 19:48:27.410770 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dbfe837064fb41238410f42ada6b39a7" } modify_external_catalogs: true
I20251024 19:48:27.410949 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dbfe837064fb41238410f42ada6b39a7" } modify_external_catalogs: true
I20251024 19:48:27.411553 1774 ts_tablet_manager.cc:1929] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:27.411751 1774 log.cc:1199] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/80c58e2439dc490c824b5f0023b5434b
I20251024 19:48:27.412179 1774 ts_tablet_manager.cc:1950] T 80c58e2439dc490c824b5f0023b5434b P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:27.413319 1768 tablet_bootstrap.cc:492] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.413512 1768 ts_tablet_manager.cc:1403] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.064s user 0.011s sys 0.039s
I20251024 19:48:27.414280 1768 raft_consensus.cc:359] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.414505 1768 raft_consensus.cc:385] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.414597 1768 raft_consensus.cc:740] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.414749 3607 raft_consensus.cc:493] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.414845 1768 consensus_queue.cc:260] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.414877 3607 raft_consensus.cc:515] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.415388 1768 ts_tablet_manager.cc:1434] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:27.415445 3607 leader_election.cc:290] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.416069 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa08555dbe1e4ad8968c534510eca9ff" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:27.416525 1447 leader_election.cc:343] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.417449 3466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 19:48:27.426151 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 80c58e2439dc490c824b5f0023b5434b (table default.Table_2aa9405152c14d538c7164b50b336c7f [id=c00489e309ba4ada921f101a95bc8b48]) successfully deleted
I20251024 19:48:27.426259 1774 tablet_replica.cc:333] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:27.426443 1774 raft_consensus.cc:2243] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:27.426436 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3e65583d0ea84a3ea10ae11462eac663 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e65583d0ea84a3ea10ae11462eac663 already in progress: creating tablet
I20251024 19:48:27.426604 1774 raft_consensus.cc:2272] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.427048 1768 tablet_bootstrap.cc:492] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:27.427372 1774 ts_tablet_manager.cc:1916] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.428691 1768 tablet_bootstrap.cc:654] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.434237 1774 ts_tablet_manager.cc:1929] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:27.434554 1774 log.cc:1199] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/769f576810ab49368eb0fc7837b7da58
I20251024 19:48:27.434924 1774 ts_tablet_manager.cc:1950] T 769f576810ab49368eb0fc7837b7da58 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:27.435442 1766 tablet_bootstrap.cc:492] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:27.435598 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 769f576810ab49368eb0fc7837b7da58 (table default.Table_2a9850f5d5dc499f87e87bdb0efcc93a [id=1da47e6010f548d1b3d8c538150a8a8e]) successfully deleted
I20251024 19:48:27.437132 1766 tablet_bootstrap.cc:654] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.439456 1768 tablet_bootstrap.cc:492] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.439596 1768 ts_tablet_manager.cc:1403] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20251024 19:48:27.440378 1768 raft_consensus.cc:359] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.440546 1768 raft_consensus.cc:385] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.440595 1768 raft_consensus.cc:740] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.440822 1768 consensus_queue.cc:260] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.441309 1768 ts_tablet_manager.cc:1434] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:27.441597 1768 tablet_bootstrap.cc:492] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:27.443199 1768 tablet_bootstrap.cc:654] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.449987 3466 hybrid_clock.cc:648] HybridClock initialized: now 1761335307445351 us; error 4617 us; skew 500 ppm
I20251024 19:48:27.450507 3466 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251024 19:48:27.453538 1766 tablet_bootstrap.cc:492] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:27.453735 1766 ts_tablet_manager.cc:1403] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20251024 19:48:27.454439 1766 raft_consensus.cc:359] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.454613 1766 raft_consensus.cc:385] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.454668 1766 raft_consensus.cc:740] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.454881 1766 consensus_queue.cc:260] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.455982 1766 ts_tablet_manager.cc:1434] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:27.456565 1766 tablet_bootstrap.cc:492] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:27.458392 1766 tablet_bootstrap.cc:654] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.458449 1768 tablet_bootstrap.cc:492] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.458573 1768 ts_tablet_manager.cc:1403] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20251024 19:48:27.460494 1768 raft_consensus.cc:359] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.460662 1768 raft_consensus.cc:385] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.460727 1768 raft_consensus.cc:740] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.460927 1768 consensus_queue.cc:260] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.462025 1768 ts_tablet_manager.cc:1434] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:27.469223 3607 raft_consensus.cc:493] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.469354 3607 raft_consensus.cc:515] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.469937 3607 leader_election.cc:290] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.471037 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d1b6eaa353c4284a2b114b42437272c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.471038 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46c76d86a68c40e18cc176b4776db266" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:27.471472 1447 leader_election.cc:343] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.479569 1775 ts_tablet_manager.cc:1929] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:27.479696 1775 log.cc:1199] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/edb49884389c4dba957f0ee5af4ae0e4
I20251024 19:48:27.480091 1775 ts_tablet_manager.cc:1950] T edb49884389c4dba957f0ee5af4ae0e4 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:27.480829 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1065) took 5686 ms. Trace:
I20251024 19:48:27.480898 1775 rpcz_store.cc:276] 1024 19:48:21.794083 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.797069 (+ 2986us) service_pool.cc:225] Handling call
1024 19:48:27.301813 (+5504744us) ts_tablet_manager.cc:1151] Deleting tablet edb49884389c4dba957f0ee5af4ae0e4
1024 19:48:27.301821 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.384132 (+ 82311us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.479432 (+ 95300us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.480809 (+ 1377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5504514}
I20251024 19:48:27.481251 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1064) took 5687 ms. Trace:
I20251024 19:48:27.482370 1775 rpcz_store.cc:276] 1024 19:48:21.794011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.797623 (+ 3612us) service_pool.cc:225] Handling call
1024 19:48:27.481135 (+5683512us) ts_tablet_manager.cc:1151] Deleting tablet 4ddce416119e4f08b81be4b5e6760144
1024 19:48:27.481143 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.481238 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5683282}
I20251024 19:48:27.482792 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1067) took 5617 ms. Trace:
I20251024 19:48:27.483109 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet edb49884389c4dba957f0ee5af4ae0e4 (table default.Table_07aae1757a3444a4bbb8d729d6f08a1d [id=b410120363fe4ba5907be2c8793b5e8c]) successfully deleted
I20251024 19:48:27.483091 1775 rpcz_store.cc:276] 1024 19:48:21.864856 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.864970 (+ 114us) service_pool.cc:225] Handling call
1024 19:48:27.482686 (+5617716us) ts_tablet_manager.cc:1151] Deleting tablet c89fa8932dbb4f4fbf7fd9f57bff27c3
1024 19:48:27.482694 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.482778 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5600677}
W20251024 19:48:27.483413 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4ddce416119e4f08b81be4b5e6760144 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ddce416119e4f08b81be4b5e6760144 already in progress: creating tablet
I20251024 19:48:27.483594 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1069) took 5514 ms. Trace:
W20251024 19:48:27.483690 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet c89fa8932dbb4f4fbf7fd9f57bff27c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c89fa8932dbb4f4fbf7fd9f57bff27c3 already in progress: creating tablet
I20251024 19:48:27.483718 1775 rpcz_store.cc:276] 1024 19:48:21.969286 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.969342 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:27.483465 (+5514123us) ts_tablet_manager.cc:1151] Deleting tablet 3cf4596b333244cc9e9e35a128721e4a
1024 19:48:27.483473 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.483580 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5513926}
W20251024 19:48:27.484081 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3cf4596b333244cc9e9e35a128721e4a because the tablet was not found. No further retry: Not found: Tablet not found: 3cf4596b333244cc9e9e35a128721e4a
I20251024 19:48:27.484129 1775 tablet_replica.cc:333] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:27.484395 1775 raft_consensus.cc:2243] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.484572 1775 raft_consensus.cc:2272] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.485080 1775 ts_tablet_manager.cc:1916] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.485270 1644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa08555dbe1e4ad8968c534510eca9ff" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:27.485595 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ef2add387da44868dbd2b5724ee7d86" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:27.485625 1645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46c76d86a68c40e18cc176b4776db266" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:27.486999 1444 leader_election.cc:343] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.487105 1444 leader_election.cc:304] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:27.487463 1444 leader_election.cc:343] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.487546 1444 leader_election.cc:304] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:27.487902 1444 leader_election.cc:343] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.487986 1444 leader_election.cc:304] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.489058 3538 raft_consensus.cc:2749] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:27.489233 3538 raft_consensus.cc:2749] T 46c76d86a68c40e18cc176b4776db266 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:27.489336 3538 raft_consensus.cc:2749] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:27.496263 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1205) took 2521 ms. Trace:
I20251024 19:48:27.496393 1776 rpcz_store.cc:276] 1024 19:48:24.974601 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.974888 (+ 287us) service_pool.cc:225] Handling call
1024 19:48:27.394804 (+2419916us) ts_tablet_manager.cc:1151] Deleting tablet 24ff2839ff1e48d5a2f84890ec25c75b
1024 19:48:27.496112 (+101308us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.496236 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2419710}
I20251024 19:48:27.496778 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1208) took 2485 ms. Trace:
I20251024 19:48:27.496869 1776 rpcz_store.cc:276] 1024 19:48:25.010936 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.016655 (+ 5719us) service_pool.cc:225] Handling call
1024 19:48:27.496679 (+2480024us) ts_tablet_manager.cc:1151] Deleting tablet 018d3e6aba6740f0bd82e8c83863bc4e
1024 19:48:27.496689 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.496763 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2404294}
W20251024 19:48:27.497756 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 24ff2839ff1e48d5a2f84890ec25c75b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ff2839ff1e48d5a2f84890ec25c75b already in progress: creating tablet
W20251024 19:48:27.497980 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 018d3e6aba6740f0bd82e8c83863bc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 018d3e6aba6740f0bd82e8c83863bc4e already in progress: creating tablet
I20251024 19:48:27.498153 1767 tablet_bootstrap.cc:492] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:27.501536 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1211) took 2451 ms. Trace:
I20251024 19:48:27.501682 1776 rpcz_store.cc:276] 1024 19:48:25.050233 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.050683 (+ 450us) service_pool.cc:225] Handling call
1024 19:48:27.497062 (+2446379us) ts_tablet_manager.cc:1151] Deleting tablet 6c887b0df24c45d894ca10d2eda71444
1024 19:48:27.501361 (+ 4299us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.501509 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2404337}
I20251024 19:48:27.502099 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1209) took 2471 ms. Trace:
I20251024 19:48:27.502189 1776 rpcz_store.cc:276] 1024 19:48:25.030879 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.031609 (+ 730us) service_pool.cc:225] Handling call
1024 19:48:27.501969 (+2470360us) ts_tablet_manager.cc:1151] Deleting tablet 7e747880c4e1425ba749201e624a083e
1024 19:48:27.501982 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.502079 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2405352}
I20251024 19:48:27.502557 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1210) took 2454 ms. Trace:
I20251024 19:48:27.502645 1776 rpcz_store.cc:276] 1024 19:48:25.048559 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.051862 (+ 3303us) service_pool.cc:225] Handling call
1024 19:48:27.502433 (+2450571us) ts_tablet_manager.cc:1151] Deleting tablet 8f46974e56c64a58a9c562b235ff8cfe
1024 19:48:27.502445 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.502538 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393592}
I20251024 19:48:27.502951 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1223) took 2076 ms. Trace:
I20251024 19:48:27.503051 1776 rpcz_store.cc:276] 1024 19:48:25.426915 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.438142 (+ 11227us) service_pool.cc:225] Handling call
1024 19:48:27.502856 (+2064714us) ts_tablet_manager.cc:1151] Deleting tablet 5ace18d554f14acc892bc956c30e9979
1024 19:48:27.502864 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.502932 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2063686}
I20251024 19:48:27.503392 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1225) took 2065 ms. Trace:
I20251024 19:48:27.503527 1776 rpcz_store.cc:276] 1024 19:48:25.437708 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.442010 (+ 4302us) service_pool.cc:225] Handling call
1024 19:48:27.503277 (+2061267us) ts_tablet_manager.cc:1151] Deleting tablet 977b4070419948d4910e2b69e01ff132
1024 19:48:27.503288 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.503374 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2061049}
I20251024 19:48:27.503801 1776 tablet_replica.cc:333] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:27.504077 1776 raft_consensus.cc:2243] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:27.504518 1776 raft_consensus.cc:2272] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.504631 1775 ts_tablet_manager.cc:1929] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251024 19:48:27.504920 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 6c887b0df24c45d894ca10d2eda71444 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c887b0df24c45d894ca10d2eda71444 already in progress: creating tablet
W20251024 19:48:27.505177 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 7e747880c4e1425ba749201e624a083e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e747880c4e1425ba749201e624a083e already in progress: creating tablet
W20251024 19:48:27.505359 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 8f46974e56c64a58a9c562b235ff8cfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f46974e56c64a58a9c562b235ff8cfe already in progress: creating tablet
W20251024 19:48:27.505548 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 5ace18d554f14acc892bc956c30e9979 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ace18d554f14acc892bc956c30e9979 already in progress: creating tablet
W20251024 19:48:27.505756 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 977b4070419948d4910e2b69e01ff132 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 977b4070419948d4910e2b69e01ff132 already in progress: creating tablet
I20251024 19:48:27.507772 1775 log.cc:1199] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/ee3c521876324808857f7724028c895f
I20251024 19:48:27.508150 1775 ts_tablet_manager.cc:1950] T ee3c521876324808857f7724028c895f P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:27.508951 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1070) took 5539 ms. Trace:
I20251024 19:48:27.509088 1775 rpcz_store.cc:276] 1024 19:48:21.969859 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:21.969907 (+ 48us) service_pool.cc:225] Handling call
1024 19:48:27.484090 (+5514183us) ts_tablet_manager.cc:1151] Deleting tablet ee3c521876324808857f7724028c895f
1024 19:48:27.484098 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.489172 (+ 5074us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.504521 (+ 15349us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.508933 (+ 4412us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5514033}
I20251024 19:48:27.509624 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet ee3c521876324808857f7724028c895f (table default.Table_a86db858d09749a9a4706c761c20ee9a [id=f49bfd0c428d44f69a3d97a6c51a380a]) successfully deleted
I20251024 19:48:27.509997 1775 tablet_replica.cc:333] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:27.510260 1775 raft_consensus.cc:2243] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.510308 1767 tablet_bootstrap.cc:654] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.511771 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_435492c6b1164482acacb1acbaa70afd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:27.512372 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_435492c6b1164482acacb1acbaa70afd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.513172 1775 raft_consensus.cc:2272] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.513931 1775 ts_tablet_manager.cc:1916] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.514377 1776 ts_tablet_manager.cc:1916] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:27.525786 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 489752031a5a4d168367cb4d143702b7 consensus state has no leader
I20251024 19:48:27.529186 1775 ts_tablet_manager.cc:1929] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:27.529649 1775 log.cc:1199] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/aa08555dbe1e4ad8968c534510eca9ff
I20251024 19:48:27.537971 1775 ts_tablet_manager.cc:1950] T aa08555dbe1e4ad8968c534510eca9ff P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:27.538079 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d61d046099f34d0284dcee14b932fc2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:27.538756 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d61d046099f34d0284dcee14b932fc2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.539216 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1071) took 5416 ms. Trace:
I20251024 19:48:27.539353 1775 rpcz_store.cc:276] 1024 19:48:22.122796 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.122850 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:27.509935 (+5387085us) ts_tablet_manager.cc:1151] Deleting tablet aa08555dbe1e4ad8968c534510eca9ff
1024 19:48:27.509944 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.521702 (+ 11758us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.529053 (+ 7351us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.539190 (+ 10137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5386775}
I20251024 19:48:27.539928 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet aa08555dbe1e4ad8968c534510eca9ff (table default.Table_b33dc95780534701ab219c111a03ee38 [id=79c88afabb0d4729a959acc2c39ef387]) successfully deleted
I20251024 19:48:27.540373 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1072) took 5361 ms. Trace:
I20251024 19:48:27.540499 1775 rpcz_store.cc:276] 1024 19:48:22.178633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.178687 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:27.540236 (+5361549us) ts_tablet_manager.cc:1151] Deleting tablet ce374bf694604250b3bbef15856ad01d
1024 19:48:27.540246 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.540355 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5361258}
W20251024 19:48:27.541002 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet ce374bf694604250b3bbef15856ad01d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce374bf694604250b3bbef15856ad01d already in progress: creating tablet
I20251024 19:48:27.541429 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1073) took 5324 ms. Trace:
I20251024 19:48:27.541558 1775 rpcz_store.cc:276] 1024 19:48:22.217015 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.217092 (+ 77us) service_pool.cc:225] Handling call
1024 19:48:27.541323 (+5324231us) ts_tablet_manager.cc:1151] Deleting tablet d9367428428648f7ae07722ea09d70a7
1024 19:48:27.541332 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.541413 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5323981}
W20251024 19:48:27.542179 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet d9367428428648f7ae07722ea09d70a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9367428428648f7ae07722ea09d70a7 already in progress: creating tablet
I20251024 19:48:27.542575 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1074) took 4814 ms. Trace:
I20251024 19:48:27.542716 1775 rpcz_store.cc:276] 1024 19:48:22.727636 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.731162 (+ 3526us) service_pool.cc:225] Handling call
1024 19:48:27.542473 (+4811311us) ts_tablet_manager.cc:1151] Deleting tablet d840888183ec4ef7bb245363c5bac322
1024 19:48:27.542481 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.542551 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4811095}
W20251024 19:48:27.543162 1179 catalog_manager.cc:4952] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet d840888183ec4ef7bb245363c5bac322 because the tablet was not found. No further retry: Not found: Tablet not found: d840888183ec4ef7bb245363c5bac322
I20251024 19:48:27.543494 1775 tablet_replica.cc:333] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:27.543808 1775 raft_consensus.cc:2243] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.544032 1775 raft_consensus.cc:2272] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.544739 1776 ts_tablet_manager.cc:1929] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:27.544898 1776 log.cc:1199] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/4100abc9ff12471e9a4959b6b24f7e99
I20251024 19:48:27.544914 1775 ts_tablet_manager.cc:1916] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.545374 1776 ts_tablet_manager.cc:1950] T 4100abc9ff12471e9a4959b6b24f7e99 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:27.545658 1766 tablet_bootstrap.cc:492] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:27.545789 1766 ts_tablet_manager.cc:1403] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.000s
I20251024 19:48:27.546465 1766 raft_consensus.cc:359] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.546644 1766 raft_consensus.cc:385] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.546698 1766 raft_consensus.cc:740] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
W20251024 19:48:27.548482 1444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 60582.7us
I20251024 19:48:27.557874 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1d23448dda3547c0a40cc1abbbd4ffc9" } modify_external_catalogs: true
I20251024 19:48:27.558100 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1d23448dda3547c0a40cc1abbbd4ffc9" } modify_external_catalogs: true
I20251024 19:48:27.558431 1775 ts_tablet_manager.cc:1929] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:27.558552 1775 log.cc:1199] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/7be9290daabb404f955f05922d897fbf
I20251024 19:48:27.559062 1775 ts_tablet_manager.cc:1950] T 7be9290daabb404f955f05922d897fbf P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:27.560034 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1075) took 4832 ms. Trace:
I20251024 19:48:27.560197 1775 rpcz_store.cc:276] 1024 19:48:22.727732 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.731426 (+ 3694us) service_pool.cc:225] Handling call
1024 19:48:27.543443 (+4812017us) ts_tablet_manager.cc:1151] Deleting tablet 7be9290daabb404f955f05922d897fbf
1024 19:48:27.543453 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.549392 (+ 5939us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.558279 (+ 8887us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.560007 (+ 1728us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4811843}
I20251024 19:48:27.560824 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 7be9290daabb404f955f05922d897fbf (table default.Table_65097482f924409b87d7f4dcb5029849 [id=867c6e52ff2f4eb3a45495738770709d]) successfully deleted
I20251024 19:48:27.561249 1775 tablet_replica.cc:333] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:27.561553 1775 raft_consensus.cc:2243] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.562796 1775 raft_consensus.cc:2272] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.563463 1775 ts_tablet_manager.cc:1916] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.563959 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1222) took 2137 ms. Trace:
I20251024 19:48:27.564191 1776 rpcz_store.cc:276] 1024 19:48:25.426825 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.447324 (+ 20499us) service_pool.cc:225] Handling call
1024 19:48:27.503746 (+2056422us) ts_tablet_manager.cc:1151] Deleting tablet 4100abc9ff12471e9a4959b6b24f7e99
1024 19:48:27.503757 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.540076 (+ 36319us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.544601 (+ 4525us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.563896 (+ 19295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2056237}
I20251024 19:48:27.565840 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 4100abc9ff12471e9a4959b6b24f7e99 (table default.Table_da24358ca9544a0e805bb937715888f9 [id=0e3c3f6ff360497c93fac944f351702c]) successfully deleted
I20251024 19:48:27.567742 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6a973600640545a6947cf458bf8ca233" } modify_external_catalogs: true
I20251024 19:48:27.568061 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6a973600640545a6947cf458bf8ca233" } modify_external_catalogs: true
I20251024 19:48:27.574091 3599 raft_consensus.cc:493] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.574252 3599 raft_consensus.cc:515] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.574851 3599 leader_election.cc:290] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:27.575294 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1fc0f99ca5c42bfb9663e21a7b761c9" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:27.576030 1305 leader_election.cc:343] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:27.581837 1586 consensus_peers.cc:597] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:27.582054 3623 raft_consensus.cc:493] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.582353 3623 raft_consensus.cc:515] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.583518 3623 leader_election.cc:290] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.584306 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2671108031db449cae2135835fc5f739" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:27.584991 1447 leader_election.cc:343] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.585036 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2671108031db449cae2135835fc5f739" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:27.585924 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1fc0f99ca5c42bfb9663e21a7b761c9" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:27.586175 1508 raft_consensus.cc:2468] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:27.587083 1308 leader_election.cc:304] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.587471 3599 raft_consensus.cc:2804] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:27.587563 3599 raft_consensus.cc:493] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:27.587625 3599 raft_consensus.cc:3060] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:27.581502 1775 ts_tablet_manager.cc:1929] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251024 19:48:27.588097 1444 leader_election.cc:343] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.588101 1775 log.cc:1199] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/73278db2097d4001a538d62dacb97c9b
I20251024 19:48:27.588702 1775 ts_tablet_manager.cc:1950] T 73278db2097d4001a538d62dacb97c9b P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:27.589600 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1076) took 4861 ms. Trace:
I20251024 19:48:27.590548 3599 raft_consensus.cc:515] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.591815 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1fc0f99ca5c42bfb9663e21a7b761c9" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:27.591931 3599 leader_election.cc:290] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:27.592321 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1224) took 2165 ms. Trace:
W20251024 19:48:27.592355 1305 leader_election.cc:343] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.592407 1776 rpcz_store.cc:276] 1024 19:48:25.427017 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.447545 (+ 20528us) service_pool.cc:225] Handling call
1024 19:48:27.592209 (+2144664us) ts_tablet_manager.cc:1151] Deleting tablet 28f93be6762146398421ac19d2e84e0c
1024 19:48:27.592221 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.592307 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2143668}
I20251024 19:48:27.593035 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1226) took 2138 ms. Trace:
I20251024 19:48:27.593118 1776 rpcz_store.cc:276] 1024 19:48:25.455000 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.455275 (+ 275us) service_pool.cc:225] Handling call
1024 19:48:27.592923 (+2137648us) ts_tablet_manager.cc:1151] Deleting tablet bec3806757214c869cf1c4cecd7cc26e
1024 19:48:27.592933 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.593019 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2137469}
W20251024 19:48:27.593753 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 28f93be6762146398421ac19d2e84e0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28f93be6762146398421ac19d2e84e0c already in progress: creating tablet
W20251024 19:48:27.594022 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet bec3806757214c869cf1c4cecd7cc26e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bec3806757214c869cf1c4cecd7cc26e already in progress: creating tablet
I20251024 19:48:27.595516 1775 rpcz_store.cc:276] 1024 19:48:22.727819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.731639 (+ 3820us) service_pool.cc:225] Handling call
1024 19:48:27.561186 (+4829547us) ts_tablet_manager.cc:1151] Deleting tablet 73278db2097d4001a538d62dacb97c9b
1024 19:48:27.561198 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.578394 (+ 17196us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.581396 (+ 3002us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.589554 (+ 8158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4829387}
I20251024 19:48:27.595980 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 73278db2097d4001a538d62dacb97c9b (table default.Table_5114b358c69441539cfc49077ea63f9e [id=0568912ed6824b29bf68cd4f25ca2dff]) successfully deleted
I20251024 19:48:27.588266 1444 leader_election.cc:304] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.596521 3623 raft_consensus.cc:2749] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:27.596868 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1fc0f99ca5c42bfb9663e21a7b761c9" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:27.597059 1508 raft_consensus.cc:3060] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:27.597740 1776 tablet_replica.cc:333] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:27.598003 1776 raft_consensus.cc:2243] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:27.599709 1776 raft_consensus.cc:2272] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.602289 1508 raft_consensus.cc:2468] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:27.602788 1308 leader_election.cc:304] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.603065 3599 raft_consensus.cc:2804] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:27.603169 3599 raft_consensus.cc:697] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:27.604151 3599 consensus_queue.cc:237] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.603672 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1077) took 4867 ms. Trace:
I20251024 19:48:27.605237 1775 rpcz_store.cc:276] 1024 19:48:22.736030 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.736101 (+ 71us) service_pool.cc:225] Handling call
1024 19:48:27.603532 (+4867431us) ts_tablet_manager.cc:1151] Deleting tablet 4e14a82420f24157a36df44d050b88bf
1024 19:48:27.603545 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.603654 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4867142}
W20251024 19:48:27.605899 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4e14a82420f24157a36df44d050b88bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e14a82420f24157a36df44d050b88bf already in progress: creating tablet
I20251024 19:48:27.606377 1775 tablet_replica.cc:333] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:27.606686 1775 raft_consensus.cc:2243] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.606928 1775 raft_consensus.cc:2272] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.608000 1775 ts_tablet_manager.cc:1916] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.609889 1776 ts_tablet_manager.cc:1916] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.618547 1766 consensus_queue.cc:260] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.619385 1766 ts_tablet_manager.cc:1434] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.074s user 0.002s sys 0.000s
I20251024 19:48:27.620163 1766 tablet_bootstrap.cc:492] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:27.620141 1775 ts_tablet_manager.cc:1929] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:27.621174 1775 log.cc:1199] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/780442b310e6409498e6bbe4256280a4
I20251024 19:48:27.621734 1767 tablet_bootstrap.cc:492] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:27.621918 1767 ts_tablet_manager.cc:1403] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.000s
I20251024 19:48:27.622211 1775 ts_tablet_manager.cc:1950] T 780442b310e6409498e6bbe4256280a4 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:27.622977 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1078) took 4885 ms. Trace:
I20251024 19:48:27.622952 1767 raft_consensus.cc:359] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.623136 1767 raft_consensus.cc:385] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.623190 1767 raft_consensus.cc:740] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.623392 1767 consensus_queue.cc:260] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.623845 1767 ts_tablet_manager.cc:1434] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:27.623961 3466 webserver.cc:492] Webserver started at http://127.0.252.253:43901/ using document root <none> and password file <none>
I20251024 19:48:27.624142 1767 tablet_bootstrap.cc:492] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:27.625205 3466 fs_manager.cc:362] Metadata directory not provided
I20251024 19:48:27.625433 3466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 19:48:27.626488 1775 rpcz_store.cc:276] 1024 19:48:22.737886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.737940 (+ 54us) service_pool.cc:225] Handling call
1024 19:48:27.606309 (+4868369us) ts_tablet_manager.cc:1151] Deleting tablet 780442b310e6409498e6bbe4256280a4
1024 19:48:27.606323 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.611905 (+ 5582us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.620039 (+ 8134us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.622956 (+ 2917us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4868122}
I20251024 19:48:27.627161 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 780442b310e6409498e6bbe4256280a4 (table default.Table_7966ff9879d24254aed39562b2f127d7 [id=45b38d5e9fd6472fa318b41863080708]) successfully deleted
I20251024 19:48:27.627539 1775 tablet_replica.cc:333] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:27.627776 1775 raft_consensus.cc:2243] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.627981 1775 raft_consensus.cc:2272] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.628543 1775 ts_tablet_manager.cc:1916] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.632846 1768 tablet_bootstrap.cc:492] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:27.634749 1768 tablet_bootstrap.cc:654] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.649457 1775 ts_tablet_manager.cc:1929] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:27.649860 1775 log.cc:1199] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/57f522a825264ea7b215304dbb7aa2d5
I20251024 19:48:27.650296 1775 ts_tablet_manager.cc:1950] T 57f522a825264ea7b215304dbb7aa2d5 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:27.651115 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1079) took 4899 ms. Trace:
I20251024 19:48:27.651250 1775 rpcz_store.cc:276] 1024 19:48:22.751876 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.752020 (+ 144us) service_pool.cc:225] Handling call
1024 19:48:27.627492 (+4875472us) ts_tablet_manager.cc:1151] Deleting tablet 57f522a825264ea7b215304dbb7aa2d5
1024 19:48:27.627502 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.631865 (+ 4363us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.649304 (+ 17439us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.651098 (+ 1794us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4875202}
I20251024 19:48:27.651822 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 57f522a825264ea7b215304dbb7aa2d5 (table default.Table_6a4dbd8e2b264bbca433946dfb1890b7 [id=9fec2fa31d474c5f8bc3b39041ed30bb]) successfully deleted
I20251024 19:48:27.652303 1775 tablet_replica.cc:333] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:27.652609 1775 raft_consensus.cc:2243] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.652849 1775 raft_consensus.cc:2272] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.653478 1775 ts_tablet_manager.cc:1916] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.655525 1370 raft_consensus.cc:3060] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:27.655714 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3b59a05b195a4d728ae60c0e25c5037f" } new_table_name: "default.Table_f7f2e6d903ed4b139eaae42e0dc95366" modify_external_catalogs: true
I20251024 19:48:27.659125 3466 fs_manager.cc:714] Time spent opening directory manager: real 0.031s user 0.007s sys 0.005s
I20251024 19:48:27.660065 3573 consensus_queue.cc:1048] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:27.667464 1775 ts_tablet_manager.cc:1929] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:27.667708 1775 log.cc:1199] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/ab488e4971874a16b2e052c84c250777
I20251024 19:48:27.668126 1775 ts_tablet_manager.cc:1950] T ab488e4971874a16b2e052c84c250777 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:27.669003 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1080) took 4883 ms. Trace:
I20251024 19:48:27.670349 1768 tablet_bootstrap.cc:492] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.670569 1768 ts_tablet_manager.cc:1403] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.002s
I20251024 19:48:27.670590 1775 rpcz_store.cc:276] 1024 19:48:22.785936 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.785998 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:27.652245 (+4866247us) ts_tablet_manager.cc:1151] Deleting tablet ab488e4971874a16b2e052c84c250777
1024 19:48:27.652258 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.660547 (+ 8289us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.667341 (+ 6794us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.668980 (+ 1639us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4865954}
I20251024 19:48:27.671198 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet ab488e4971874a16b2e052c84c250777 (table default.Table_93cc576cc317428c88108b05bf8efc6d [id=c69469ba6a9745f6a66092a678a5861d]) successfully deleted
I20251024 19:48:27.671540 1768 raft_consensus.cc:359] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.671725 1768 raft_consensus.cc:385] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.671782 1768 raft_consensus.cc:740] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.672037 1768 consensus_queue.cc:260] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.672576 1768 ts_tablet_manager.cc:1434] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:27.673492 1768 tablet_bootstrap.cc:492] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:27.679325 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1081) took 4882 ms. Trace:
I20251024 19:48:27.679361 3599 raft_consensus.cc:493] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.679482 3599 raft_consensus.cc:515] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.680541 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f522a825264ea7b215304dbb7aa2d5" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:27.679527 1775 rpcz_store.cc:276] 1024 19:48:22.797149 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:22.797784 (+ 635us) service_pool.cc:225] Handling call
1024 19:48:27.678077 (+4880293us) ts_tablet_manager.cc:1151] Deleting tablet 4e14a82420f24157a36df44d050b88bf
1024 19:48:27.678089 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.679298 (+ 1209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4880018}
W20251024 19:48:27.681102 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4e14a82420f24157a36df44d050b88bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e14a82420f24157a36df44d050b88bf already in progress: creating tablet
I20251024 19:48:27.681491 3599 leader_election.cc:290] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:27.681618 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1097) took 4647 ms. Trace:
I20251024 19:48:27.683806 1775 rpcz_store.cc:276] 1024 19:48:23.033992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.034062 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:27.681420 (+4647358us) ts_tablet_manager.cc:1151] Deleting tablet 80688d7b8a394a2a910c4b8910fe4cf2
1024 19:48:27.681442 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.681560 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4646984}
W20251024 19:48:27.684373 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 80688d7b8a394a2a910c4b8910fe4cf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80688d7b8a394a2a910c4b8910fe4cf2 already in progress: creating tablet
W20251024 19:48:27.685134 1308 leader_election.cc:343] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 57f522a825264ea7b215304dbb7aa2d5
I20251024 19:48:27.685240 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1098) took 4650 ms. Trace:
I20251024 19:48:27.685288 1308 leader_election.cc:304] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.685314 1775 rpcz_store.cc:276] 1024 19:48:23.034699 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.034794 (+ 95us) service_pool.cc:225] Handling call
1024 19:48:27.684726 (+4649932us) ts_tablet_manager.cc:1151] Deleting tablet 83d34ec47c2845058e870c72cc832e9e
1024 19:48:27.684736 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.685221 (+ 485us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4649670}
W20251024 19:48:27.685830 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 83d34ec47c2845058e870c72cc832e9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83d34ec47c2845058e870c72cc832e9e already in progress: creating tablet
I20251024 19:48:27.686296 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1099) took 4648 ms. Trace:
I20251024 19:48:27.686417 1775 rpcz_store.cc:276] 1024 19:48:23.038089 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.038254 (+ 165us) service_pool.cc:225] Handling call
1024 19:48:27.686194 (+4647940us) ts_tablet_manager.cc:1151] Deleting tablet 9198321bcc9e4e47a4c2a5442b6bc850
1024 19:48:27.686204 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.686281 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4647555}
W20251024 19:48:27.686741 1586 consensus_peers.cc:597] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7be9290daabb404f955f05922d897fbf. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:27.686993 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9198321bcc9e4e47a4c2a5442b6bc850 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9198321bcc9e4e47a4c2a5442b6bc850 already in progress: creating tablet
I20251024 19:48:27.687139 3599 raft_consensus.cc:2749] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:27.687422 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1100) took 4648 ms. Trace:
I20251024 19:48:27.689750 1187 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_45d12e1dd63849c783278f2435d7c3a6 [id=2ae1c808837e48df97c4efb01a684da9] alter complete (version 2)
I20251024 19:48:27.689900 1775 rpcz_store.cc:276] 1024 19:48:23.039247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.039299 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:27.687322 (+4648023us) ts_tablet_manager.cc:1151] Deleting tablet 9b7ab1bf5b9549afb150802a209527eb
1024 19:48:27.687330 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.687405 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4647772}
W20251024 19:48:27.690382 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9b7ab1bf5b9549afb150802a209527eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b7ab1bf5b9549afb150802a209527eb already in progress: creating tablet
I20251024 19:48:27.690789 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1108) took 4562 ms. Trace:
I20251024 19:48:27.690907 1775 rpcz_store.cc:276] 1024 19:48:23.128234 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.128516 (+ 282us) service_pool.cc:225] Handling call
1024 19:48:27.690682 (+4562166us) ts_tablet_manager.cc:1151] Deleting tablet cb1f850b3e104707a53d540ad5752d4d
1024 19:48:27.690692 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.690772 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4561956}
I20251024 19:48:27.691241 3633 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.003s user 0.000s sys 0.000s
W20251024 19:48:27.691681 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet cb1f850b3e104707a53d540ad5752d4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb1f850b3e104707a53d540ad5752d4d already in progress: creating tablet
I20251024 19:48:27.691875 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet c1fc0f99ca5c42bfb9663e21a7b761c9 (table default.Table_d7da4325d747469b86ff66b85690867d [id=a32bd550682746f0a2861ff8be404992]). Expected version 1 got 0
I20251024 19:48:27.692184 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1110) took 4416 ms. Trace:
I20251024 19:48:27.693315 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 73278db2097d4001a538d62dacb97c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.693413 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ac33dd71eb504d6a8d92fef51601721e" } modify_external_catalogs: true
I20251024 19:48:27.693861 1774 tablet_replica.cc:333] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:27.693816 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ac33dd71eb504d6a8d92fef51601721e" } modify_external_catalogs: true
I20251024 19:48:27.694876 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 83b91a5d764449d5b82ef6d1ee0360f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.695115 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.695288 1775 rpcz_store.cc:276] 1024 19:48:23.275306 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.275392 (+ 86us) service_pool.cc:225] Handling call
1024 19:48:27.692082 (+4416690us) ts_tablet_manager.cc:1151] Deleting tablet b24a3a820c9b496dbf6955b399c6ea8c
1024 19:48:27.692091 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.692168 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4416499}
W20251024 19:48:27.695863 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b24a3a820c9b496dbf6955b399c6ea8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b24a3a820c9b496dbf6955b399c6ea8c already in progress: creating tablet
I20251024 19:48:27.696275 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1120) took 4362 ms. Trace:
I20251024 19:48:27.696504 1775 rpcz_store.cc:276] 1024 19:48:23.336338 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.336417 (+ 79us) service_pool.cc:225] Handling call
1024 19:48:27.696151 (+4359734us) ts_tablet_manager.cc:1151] Deleting tablet 830c8469331940fc85aaad58c9085e35
1024 19:48:27.696163 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.696258 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4336536}
I20251024 19:48:27.697811 1768 tablet_bootstrap.cc:654] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.698148 3466 fs_manager.cc:730] Time spent opening block manager: real 0.037s user 0.006s sys 0.008s
I20251024 19:48:27.698366 3466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data,/tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/wal
uuid: "82e82847f8904b59b03f446b7fce6233"
format_stamp: "Formatted at 2025-10-24 19:47:52 on dist-test-slave-53sn"
I20251024 19:48:27.699568 3466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20251024 19:48:27.700335 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 830c8469331940fc85aaad58c9085e35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 830c8469331940fc85aaad58c9085e35 already in progress: creating tablet
I20251024 19:48:27.701717 1766 tablet_bootstrap.cc:654] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.702183 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1121) took 4365 ms. Trace:
I20251024 19:48:27.702302 1775 rpcz_store.cc:276] 1024 19:48:23.336787 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.336901 (+ 114us) service_pool.cc:225] Handling call
1024 19:48:27.700638 (+4363737us) ts_tablet_manager.cc:1151] Deleting tablet 4846799df5b242ce8f19c034320c8a1d
1024 19:48:27.701905 (+ 1267us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.702161 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4340900}
W20251024 19:48:27.702746 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4846799df5b242ce8f19c034320c8a1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4846799df5b242ce8f19c034320c8a1d already in progress: creating tablet
I20251024 19:48:27.703040 1775 tablet_replica.cc:333] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:27.703245 1775 raft_consensus.cc:2243] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:27.703562 1775 raft_consensus.cc:2272] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.704145 1775 ts_tablet_manager.cc:1916] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.704579 1776 ts_tablet_manager.cc:1929] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:27.704890 3599 raft_consensus.cc:493] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.705027 3599 raft_consensus.cc:515] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.706480 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58a8f4f90f794e83a158e0026e5cea7f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:27.707137 1305 leader_election.cc:343] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.707957 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58a8f4f90f794e83a158e0026e5cea7f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:27.708140 1508 raft_consensus.cc:2468] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:27.708639 1308 leader_election.cc:304] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.708930 3530 raft_consensus.cc:2804] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:27.709002 3599 leader_election.cc:290] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:27.709014 3530 raft_consensus.cc:493] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:27.709127 3530 raft_consensus.cc:3060] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:27.712553 3530 raft_consensus.cc:515] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.714241 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58a8f4f90f794e83a158e0026e5cea7f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:27.714354 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58a8f4f90f794e83a158e0026e5cea7f" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:27.714594 1508 raft_consensus.cc:3060] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:27.714756 1305 leader_election.cc:343] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.704687 1776 log.cc:1199] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/769f576810ab49368eb0fc7837b7da58
I20251024 19:48:27.715421 1776 ts_tablet_manager.cc:1950] T 769f576810ab49368eb0fc7837b7da58 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:27.717325 1508 raft_consensus.cc:2468] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:27.718082 1308 leader_election.cc:304] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.718519 3530 leader_election.cc:290] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:27.718863 3530 raft_consensus.cc:2804] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:27.719035 3530 raft_consensus.cc:697] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:27.719383 3530 consensus_queue.cc:237] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.722422 1767 tablet_bootstrap.cc:654] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.728564 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet b95e41141f5f4a5bbc39fb977d82252b
W20251024 19:48:27.729039 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b39e014607b4437b255c26c92c71861 consensus state has no leader
I20251024 19:48:27.729497 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_303b0da74b284bc5a810b08b52f820ba" } modify_external_catalogs: true
I20251024 19:48:27.729591 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d257560a020c4081a4f874b3731b7921"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:27.729693 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_303b0da74b284bc5a810b08b52f820ba" } modify_external_catalogs: true
I20251024 19:48:27.729964 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet b95e41141f5f4a5bbc39fb977d82252b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:27.730293 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d257560a020c4081a4f874b3731b7921 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.731372 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d3ffdf5cbfd84551b1e314d4a772be1a" } new_table_name: "default.Table_8514e876be154443ad1f2827119421c5" modify_external_catalogs: true
I20251024 19:48:27.732259 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_21b929f19b444807ba23a7a062df8e34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:27.732682 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21b929f19b444807ba23a7a062df8e34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:27.733799 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f73069b9201449d08a4f7b246444e891 consensus state has no leader
I20251024 19:48:27.734236 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3b4c85818a374983960d559f0e2612e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:27.734496 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 80c58e2439dc490c824b5f0023b5434b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:27.734665 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b4c85818a374983960d559f0e2612e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.734688 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 57f522a825264ea7b215304dbb7aa2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.736904 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet b95e41141f5f4a5bbc39fb977d82252b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.737072 1774 raft_consensus.cc:2243] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.737380 1774 pending_rounds.cc:70] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:27.737421 1774 pending_rounds.cc:77] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429415694053376 op_type: NO_OP noop_request { }
I20251024 19:48:27.737509 1774 raft_consensus.cc:2889] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:27.737601 1774 raft_consensus.cc:2272] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.738651 1774 ts_tablet_manager.cc:1916] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.741205 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet b95e41141f5f4a5bbc39fb977d82252b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.744091 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f405fc8974e243af982f30e00607d14a
I20251024 19:48:27.744963 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_941306808730402d8ac4181a7a4877f2" } new_table_name: "default.Table_6b63c138ce4648ad9bdfac7f05ffdea9" modify_external_catalogs: true
I20251024 19:48:27.745100 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet f405fc8974e243af982f30e00607d14a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.745725 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet f405fc8974e243af982f30e00607d14a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.747399 1774 ts_tablet_manager.cc:1929] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:27.747571 1774 log.cc:1199] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/73278db2097d4001a538d62dacb97c9b
I20251024 19:48:27.748003 1774 ts_tablet_manager.cc:1950] T 73278db2097d4001a538d62dacb97c9b P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:27.748001 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet f405fc8974e243af982f30e00607d14a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.748236 1766 tablet_bootstrap.cc:492] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:27.748406 1766 ts_tablet_manager.cc:1403] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.128s user 0.003s sys 0.001s
I20251024 19:48:27.748759 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 73278db2097d4001a538d62dacb97c9b (table default.Table_5114b358c69441539cfc49077ea63f9e [id=0568912ed6824b29bf68cd4f25ca2dff]) successfully deleted
I20251024 19:48:27.748872 1768 tablet_bootstrap.cc:492] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.748992 1768 ts_tablet_manager.cc:1403] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.076s user 0.001s sys 0.003s
I20251024 19:48:27.749153 1774 tablet_replica.cc:333] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:27.749343 1774 raft_consensus.cc:2243] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.749265 1766 raft_consensus.cc:359] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.749446 1766 raft_consensus.cc:385] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.749490 1774 raft_consensus.cc:2272] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.749511 1766 raft_consensus.cc:740] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.749889 1766 consensus_queue.cc:260] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.750223 1768 raft_consensus.cc:359] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.750448 1768 raft_consensus.cc:385] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.750551 1768 raft_consensus.cc:740] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.750777 1766 ts_tablet_manager.cc:1434] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:27.751186 1768 consensus_queue.cc:260] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.752224 1768 ts_tablet_manager.cc:1434] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251024 19:48:27.753693 1766 tablet_bootstrap.cc:492] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:27.753881 1768 tablet_bootstrap.cc:492] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:27.755513 1766 tablet_bootstrap.cc:654] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.755791 1188 master_service.cc:954] ReplaceTablet: received request to replace tablet 404c38f8db2f43bfbc63e9ba2cee902c from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:27.756783 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet be21ca442dee4204a8b15f5a2dd9530b
I20251024 19:48:27.757442 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_23181a343624498bbcf28a5ab6b564eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:27.758123 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23181a343624498bbcf28a5ab6b564eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.760046 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet be21ca442dee4204a8b15f5a2dd9530b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:27.761268 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet be21ca442dee4204a8b15f5a2dd9530b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:27.761730 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet be21ca442dee4204a8b15f5a2dd9530b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:27.769367 1775 ts_tablet_manager.cc:1929] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:27.750239 1774 ts_tablet_manager.cc:1916] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.737632 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1237) took 2123 ms. Trace:
I20251024 19:48:27.771229 1776 rpcz_store.cc:276] 1024 19:48:25.614416 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.614478 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:27.597664 (+1983186us) ts_tablet_manager.cc:1151] Deleting tablet 769f576810ab49368eb0fc7837b7da58
1024 19:48:27.597679 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.636684 (+ 39005us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.704458 (+ 67774us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.737537 (+ 33079us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1982971}
W20251024 19:48:27.770920 1308 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 44221.3us
I20251024 19:48:27.771914 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 769f576810ab49368eb0fc7837b7da58 (table default.Table_2a9850f5d5dc499f87e87bdb0efcc93a [id=1da47e6010f548d1b3d8c538150a8a8e]) successfully deleted
I20251024 19:48:27.775701 1767 tablet_bootstrap.cc:492] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:27.775904 1767 ts_tablet_manager.cc:1403] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.152s user 0.003s sys 0.000s
I20251024 19:48:27.776968 1767 raft_consensus.cc:359] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.777218 1767 raft_consensus.cc:385] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.777288 1767 raft_consensus.cc:740] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.777648 1767 consensus_queue.cc:260] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.782343 1767 ts_tablet_manager.cc:1434] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251024 19:48:27.782720 1767 tablet_bootstrap.cc:492] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:27.784878 1767 tablet_bootstrap.cc:654] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.809561 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1245) took 2053 ms. Trace:
I20251024 19:48:27.809785 1776 rpcz_store.cc:276] 1024 19:48:25.756076 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.756433 (+ 357us) service_pool.cc:225] Handling call
1024 19:48:27.809374 (+2052941us) ts_tablet_manager.cc:1151] Deleting tablet c084db163d874639b4385f41e9efaa18
1024 19:48:27.809391 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.809532 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2052732}
W20251024 19:48:27.810312 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet c084db163d874639b4385f41e9efaa18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c084db163d874639b4385f41e9efaa18 already in progress: creating tablet
I20251024 19:48:27.810750 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1246) took 2041 ms. Trace:
I20251024 19:48:27.810876 1776 rpcz_store.cc:276] 1024 19:48:25.769020 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.769520 (+ 500us) service_pool.cc:225] Handling call
1024 19:48:27.810643 (+2041123us) ts_tablet_manager.cc:1151] Deleting tablet d72f4761820e430cb1a97d83cd30c75d
1024 19:48:27.810654 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.810732 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2040873}
W20251024 19:48:27.811384 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d72f4761820e430cb1a97d83cd30c75d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d72f4761820e430cb1a97d83cd30c75d already in progress: creating tablet
I20251024 19:48:27.811779 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1248) took 2042 ms. Trace:
I20251024 19:48:27.811887 1776 rpcz_store.cc:276] 1024 19:48:25.769224 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.769920 (+ 696us) service_pool.cc:225] Handling call
1024 19:48:27.811668 (+2041748us) ts_tablet_manager.cc:1151] Deleting tablet b383a83425654723bd1d9c3a39b187be
1024 19:48:27.811678 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.811763 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2041632}
W20251024 19:48:27.812297 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b383a83425654723bd1d9c3a39b187be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b383a83425654723bd1d9c3a39b187be already in progress: creating tablet
I20251024 19:48:27.812661 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1249) took 2043 ms. Trace:
I20251024 19:48:27.812770 1776 rpcz_store.cc:276] 1024 19:48:25.769315 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.770063 (+ 748us) service_pool.cc:225] Handling call
1024 19:48:27.812567 (+2042504us) ts_tablet_manager.cc:1151] Deleting tablet e0228d5b788e4c42bf4ecce876ecc3f4
1024 19:48:27.812576 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.812646 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2042383}
W20251024 19:48:27.813174 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet e0228d5b788e4c42bf4ecce876ecc3f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0228d5b788e4c42bf4ecce876ecc3f4 already in progress: creating tablet
I20251024 19:48:27.814615 1768 tablet_bootstrap.cc:654] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.816267 1774 ts_tablet_manager.cc:1929] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251024 19:48:27.819824 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2671108031db449cae2135835fc5f739 consensus state has no leader
I20251024 19:48:27.823235 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c50cee70d21d437b917b616ea7cdddcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:27.823505 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 83b91a5d764449d5b82ef6d1ee0360f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:27.823848 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c50cee70d21d437b917b616ea7cdddcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.827855 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_2549427b0e704654a43073c7ef97be52" } new_table_name: "default.Table_f97263739394418d85ec7e1bc0d54265" modify_external_catalogs: true
I20251024 19:48:27.831557 1774 log.cc:1199] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/83b91a5d764449d5b82ef6d1ee0360f5
W20251024 19:48:27.832864 1178 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 46e5e391a7674b5d8e1dd1a780fe36c8 (table default.Table_96f18cad8ca2451e9e8a9b82da4754eb [id=e40de65cb381461bb0e5c9896a5eb812], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 46e5e391a7674b5d8e1dd1a780fe36c8 consensus state has no leader
I20251024 19:48:27.836241 1774 ts_tablet_manager.cc:1950] T 83b91a5d764449d5b82ef6d1ee0360f5 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:27.838797 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 83b91a5d764449d5b82ef6d1ee0360f5 (table default.Table_325e96b0d32e4a3aab388a916c29ec26 [id=c6b5c56913684d08b89fde458e31b8cb]) successfully deleted
W20251024 19:48:27.840180 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5282515c1844ae4b7469551436586c2 consensus state has no leader
I20251024 19:48:27.782831 3456 raft_consensus.cc:493] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.840317 1774 tablet_replica.cc:333] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:27.841812 1774 raft_consensus.cc:2243] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.841727 3456 raft_consensus.cc:515] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.841960 1774 raft_consensus.cc:2272] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.842423 3456 leader_election.cc:290] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.826014 3607 raft_consensus.cc:493] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.842621 3607 raft_consensus.cc:515] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.842903 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2f474456e9334431b6a1d56c9f4c911b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:27.843113 3607 leader_election.cc:290] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:27.843586 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f474456e9334431b6a1d56c9f4c911b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.844266 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7deb97be82964ea0b2fe9caa867a6460" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:27.844403 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "373aa543e1284fbc945ca802d9eddf59" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:27.830047 1768 tablet_bootstrap.cc:492] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.844902 1768 ts_tablet_manager.cc:1403] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.091s user 0.001s sys 0.002s
W20251024 19:48:27.844928 1447 leader_election.cc:343] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:27.845350 1444 leader_election.cc:343] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.845737 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7deb97be82964ea0b2fe9caa867a6460" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:27.846196 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "373aa543e1284fbc945ca802d9eddf59" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:27.813542 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1247) took 2044 ms. Trace:
I20251024 19:48:27.850140 1776 rpcz_store.cc:276] 1024 19:48:25.769133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.769808 (+ 675us) service_pool.cc:225] Handling call
1024 19:48:27.813436 (+2043628us) ts_tablet_manager.cc:1151] Deleting tablet 3c608ac55e9e42bc9c9b27beec717e30
1024 19:48:27.813445 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.813527 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2037167}
W20251024 19:48:27.850867 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3c608ac55e9e42bc9c9b27beec717e30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c608ac55e9e42bc9c9b27beec717e30 already in progress: creating tablet
I20251024 19:48:27.851460 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1250) took 1655 ms. Trace:
I20251024 19:48:27.851691 1776 rpcz_store.cc:276] 1024 19:48:26.196305 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.196366 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:27.851304 (+1654938us) ts_tablet_manager.cc:1151] Deleting tablet bcdd47a64e2142cda1f2d20afebd4ce5
1024 19:48:27.851318 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.851438 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654695}
I20251024 19:48:27.852267 1774 ts_tablet_manager.cc:1916] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:27.852275 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet bcdd47a64e2142cda1f2d20afebd4ce5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcdd47a64e2142cda1f2d20afebd4ce5 already in progress: creating tablet
I20251024 19:48:27.852725 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1251) took 1639 ms. Trace:
W20251024 19:48:27.846263 1444 leader_election.cc:343] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.852880 1776 rpcz_store.cc:276] 1024 19:48:26.212994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.213058 (+ 64us) service_pool.cc:225] Handling call
1024 19:48:27.852591 (+1639533us) ts_tablet_manager.cc:1151] Deleting tablet 5d62c5c336e14c65bf6734c66e9be198
1024 19:48:27.852604 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.852705 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1639314}
I20251024 19:48:27.853029 1444 leader_election.cc:304] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:27.853338 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 5d62c5c336e14c65bf6734c66e9be198 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d62c5c336e14c65bf6734c66e9be198 already in progress: creating tablet
W20251024 19:48:27.847934 1447 leader_election.cc:343] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.853468 1447 leader_election.cc:304] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.853679 3607 raft_consensus.cc:2749] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:27.853750 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1252) took 1609 ms. Trace:
I20251024 19:48:27.853838 1776 rpcz_store.cc:276] 1024 19:48:26.243896 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.243973 (+ 77us) service_pool.cc:225] Handling call
1024 19:48:27.853648 (+1609675us) ts_tablet_manager.cc:1151] Deleting tablet d2a68bb1dc6c483388784b9bb19ce065
1024 19:48:27.853661 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.853730 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1609432}
I20251024 19:48:27.854266 3607 raft_consensus.cc:2749] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:27.845558 1775 log.cc:1199] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/81f8393f78204860bd854f4872099387
W20251024 19:48:27.854435 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d2a68bb1dc6c483388784b9bb19ce065 because the tablet was not found. No further retry: Not found: Tablet not found: d2a68bb1dc6c483388784b9bb19ce065
I20251024 19:48:27.854861 1776 tablet_replica.cc:333] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:27.854931 1775 ts_tablet_manager.cc:1950] T 81f8393f78204860bd854f4872099387 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:27.855165 1776 raft_consensus.cc:2243] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.855402 1776 pending_rounds.cc:70] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:27.855515 1776 pending_rounds.cc:77] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429413847891968 op_type: NO_OP noop_request { }
I20251024 19:48:27.855702 1776 raft_consensus.cc:2889] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:27.855736 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1122) took 4516 ms. Trace:
I20251024 19:48:27.855870 1776 raft_consensus.cc:2272] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.855898 1775 rpcz_store.cc:276] 1024 19:48:23.339990 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.340483 (+ 493us) service_pool.cc:225] Handling call
1024 19:48:27.702992 (+4362509us) ts_tablet_manager.cc:1151] Deleting tablet 81f8393f78204860bd854f4872099387
1024 19:48:27.703003 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.707495 (+ 4492us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.769243 (+ 61748us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.855698 (+ 86455us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4343138}
I20251024 19:48:27.856523 3607 raft_consensus.cc:493] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.856657 3607 raft_consensus.cc:515] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.857231 3607 leader_election.cc:290] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.854020 1768 raft_consensus.cc:359] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:27.857643 1768 raft_consensus.cc:385] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.857715 1768 raft_consensus.cc:740] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.857952 1768 consensus_queue.cc:260] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:27.858172 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15de0d8aa9e545c09233186bbe361b97" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:27.858634 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15de0d8aa9e545c09233186bbe361b97" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:27.858809 1447 leader_election.cc:343] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:27.859272 1444 leader_election.cc:343] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.852869 1766 tablet_bootstrap.cc:492] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:27.859450 1444 leader_election.cc:304] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.859560 1766 ts_tablet_manager.cc:1403] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.109s user 0.001s sys 0.003s
I20251024 19:48:27.860803 1766 raft_consensus.cc:359] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:27.860984 1766 raft_consensus.cc:385] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.861063 1766 raft_consensus.cc:740] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.861557 3573 consensus_queue.cc:1048] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:27.861666 3607 raft_consensus.cc:2749] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:27.864154 1768 ts_tablet_manager.cc:1434] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20251024 19:48:27.868507 1768 tablet_bootstrap.cc:492] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
W20251024 19:48:27.868942 1586 consensus_peers.cc:597] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73278db2097d4001a538d62dacb97c9b. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:27.869371 1586 consensus_peers.cc:597] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7be9290daabb404f955f05922d897fbf. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:27.870370 1766 consensus_queue.cc:260] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:27.870954 1766 ts_tablet_manager.cc:1434] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251024 19:48:27.871251 1766 tablet_bootstrap.cc:492] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:27.872767 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_42d562f8d96045689a6825f56d7f2c10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:27.873847 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c75b9f86032e435e98f4623a0cc5edf0 has no consensus state
I20251024 19:48:27.874239 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_795726c40abd4110a8d3173989dc70d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:27.874387 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 81f8393f78204860bd854f4872099387 (table default.Table_2d74dcbcd9f249e5b10e1986cc0c0d18 [id=0bd294ee0c8341a6a29ce26caeeceb22]) successfully deleted
W20251024 19:48:27.874657 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 77487us
W20251024 19:48:27.874805 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_795726c40abd4110a8d3173989dc70d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:27.877310 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42d562f8d96045689a6825f56d7f2c10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.878113 1767 tablet_bootstrap.cc:492] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:27.878265 1767 ts_tablet_manager.cc:1403] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.000s
I20251024 19:48:27.878445 1768 tablet_bootstrap.cc:654] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.879056 1767 raft_consensus.cc:359] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.879227 1767 raft_consensus.cc:385] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.879287 1767 raft_consensus.cc:740] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.879514 1767 consensus_queue.cc:260] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.880031 1767 ts_tablet_manager.cc:1434] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:27.880331 1767 tablet_bootstrap.cc:492] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:27.882174 1767 tablet_bootstrap.cc:654] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.887306 1766 tablet_bootstrap.cc:654] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.890064 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1126) took 4438 ms. Trace:
I20251024 19:48:27.879324 1774 ts_tablet_manager.cc:1929] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:27.896287 1774 log.cc:1199] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/c1cdd5ec32ca49c1bfafaf3a24050e89
I20251024 19:48:27.896840 1774 ts_tablet_manager.cc:1950] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:27.899825 1768 tablet_bootstrap.cc:492] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.900059 1768 ts_tablet_manager.cc:1403] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20251024 19:48:27.900175 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet c1cdd5ec32ca49c1bfafaf3a24050e89 (table default.Table_7d5114465a524d37b479ed5a6cebb938 [id=fab966f6dc2c4b299c517ea170531ab8]) successfully deleted
I20251024 19:48:27.901500 1775 rpcz_store.cc:276] 1024 19:48:23.436682 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.436828 (+ 146us) service_pool.cc:225] Handling call
1024 19:48:27.874425 (+4437597us) ts_tablet_manager.cc:1151] Deleting tablet 2d8b8a1cff3540f7b7f13712425d534e
1024 19:48:27.874436 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.874549 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4369456}
W20251024 19:48:27.902136 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 2d8b8a1cff3540f7b7f13712425d534e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d8b8a1cff3540f7b7f13712425d534e already in progress: creating tablet
I20251024 19:48:27.902798 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet 7172a93a5bec4fa1b63282ba63beec0c from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:27.903301 1775 tablet_replica.cc:333] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:27.903576 1775 raft_consensus.cc:2243] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.903862 1775 raft_consensus.cc:2272] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.904421 3466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 19:48:27.904522 1775 ts_tablet_manager.cc:1916] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.901376 1768 raft_consensus.cc:359] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.908072 1768 raft_consensus.cc:385] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251024 19:48:27.908047 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet b95e41141f5f4a5bbc39fb977d82252b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b95e41141f5f4a5bbc39fb977d82252b already in progress: creating tablet
I20251024 19:48:27.908231 1768 raft_consensus.cc:740] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.908571 1768 consensus_queue.cc:260] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:27.908779 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet f405fc8974e243af982f30e00607d14a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f405fc8974e243af982f30e00607d14a already in progress: creating tablet
W20251024 19:48:27.909328 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet be21ca442dee4204a8b15f5a2dd9530b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be21ca442dee4204a8b15f5a2dd9530b already in progress: creating tablet
I20251024 19:48:27.909842 3466 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 19:48:27.911505 3466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 19:48:27.915431 1768 ts_tablet_manager.cc:1434] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251024 19:48:27.918829 1766 tablet_bootstrap.cc:492] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:27.919008 1766 ts_tablet_manager.cc:1403] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20251024 19:48:27.921269 1766 raft_consensus.cc:359] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.921608 1766 raft_consensus.cc:385] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.921682 1766 raft_consensus.cc:740] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.922024 1766 consensus_queue.cc:260] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.923404 1766 ts_tablet_manager.cc:1434] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251024 19:48:27.924679 1766 tablet_bootstrap.cc:492] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:27.925927 1776 ts_tablet_manager.cc:1916] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.932073 1768 tablet_bootstrap.cc:492] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:27.927647 1767 tablet_bootstrap.cc:492] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:27.932243 1767 ts_tablet_manager.cc:1403] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.010s
I20251024 19:48:27.933029 1767 raft_consensus.cc:359] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.933194 1767 raft_consensus.cc:385] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.933236 1767 raft_consensus.cc:740] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.933451 1767 consensus_queue.cc:260] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.933909 1767 ts_tablet_manager.cc:1434] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251024 19:48:27.934234 1767 tablet_bootstrap.cc:492] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:27.935940 1767 tablet_bootstrap.cc:654] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.936998 1766 tablet_bootstrap.cc:654] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.938369 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5783546824a846e7b084ac7e809885b9" } modify_external_catalogs: true
I20251024 19:48:27.938705 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5783546824a846e7b084ac7e809885b9" } modify_external_catalogs: true
W20251024 19:48:27.948789 1586 consensus_peers.cc:597] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:27.950734 1768 tablet_bootstrap.cc:654] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.952654 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3fba2a24fecd460fa9482068292cc326"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:27.953562 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fba2a24fecd460fa9482068292cc326 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:27.955377 3404 raft_consensus.cc:493] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.960223 1775 ts_tablet_manager.cc:1929] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:27.960371 1775 log.cc:1199] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/fc7ccf1495924782b9b6ac1173bc73a1
I20251024 19:48:27.960712 1775 ts_tablet_manager.cc:1950] T fc7ccf1495924782b9b6ac1173bc73a1 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:27.961634 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1133) took 4237 ms. Trace:
I20251024 19:48:27.961738 1775 rpcz_store.cc:276] 1024 19:48:23.724385 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.724655 (+ 270us) service_pool.cc:225] Handling call
1024 19:48:27.902641 (+4177986us) ts_tablet_manager.cc:1151] Deleting tablet fc7ccf1495924782b9b6ac1173bc73a1
1024 19:48:27.902650 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.909123 (+ 6473us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.960081 (+ 50958us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:27.961591 (+ 1510us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4177441}
I20251024 19:48:27.962097 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet fc7ccf1495924782b9b6ac1173bc73a1 (table default.Table_6a5150d7236d4d558c9f9fb4e70a78dd [id=991b0e462eba40ec882e422e60b43f26]) successfully deleted
I20251024 19:48:27.955595 3404 raft_consensus.cc:515] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:27.964510 1506 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58a8f4f90f794e83a158e0026e5cea7f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:27.965047 1506 raft_consensus.cc:2393] T 58a8f4f90f794e83a158e0026e5cea7f P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 43bec2f03fc849099e254f6335e9d532 in this term.
I20251024 19:48:27.967732 3404 leader_election.cc:290] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.969357 1775 tablet_replica.cc:333] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:27.969625 1775 raft_consensus.cc:2243] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:27.969825 1775 raft_consensus.cc:2272] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:27.970392 1775 ts_tablet_manager.cc:1916] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:27.973412 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58a8f4f90f794e83a158e0026e5cea7f" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:27.974411 1586 consensus_peers.cc:597] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc7ccf1495924782b9b6ac1173bc73a1. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:27.974803 1586 consensus_peers.cc:597] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e38ef2ac19d14f9c91fa8abe86cc7987. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:27.975126 1586 leader_election.cc:304] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:27.975575 3528 raft_consensus.cc:3060] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:27.976284 3607 raft_consensus.cc:493] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:27.976440 3607 raft_consensus.cc:515] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.977051 3607 leader_election.cc:290] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:27.977627 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0e5f7c28f0c47a88ca9872db2184ad0" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:27.978228 1444 leader_election.cc:343] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.978628 1768 tablet_bootstrap.cc:492] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:27.978663 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0e5f7c28f0c47a88ca9872db2184ad0" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:27.979202 1447 leader_election.cc:343] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:27.979321 1447 leader_election.cc:304] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:27.979722 3607 raft_consensus.cc:2749] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:27.978878 1768 ts_tablet_manager.cc:1403] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.001s
I20251024 19:48:27.980219 3528 raft_consensus.cc:2749] T 58a8f4f90f794e83a158e0026e5cea7f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:27.980832 1768 raft_consensus.cc:359] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.981058 1768 raft_consensus.cc:385] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.981163 1768 raft_consensus.cc:740] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.981462 1768 consensus_queue.cc:260] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:27.981539 1586 consensus_peers.cc:597] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee3c521876324808857f7724028c895f. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:27.985550 1768 ts_tablet_manager.cc:1434] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:48:27.989190 1768 tablet_bootstrap.cc:492] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:27.990085 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4c91e1a2322c48b69d7b31b4dbdbac64" } new_table_name: "default.Table_29b0c38a3f5340b2877ffd4a9aa8312a" modify_external_catalogs: true
I20251024 19:48:27.990988 1768 tablet_bootstrap.cc:654] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:27.996948 1766 tablet_bootstrap.cc:492] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:27.997149 1766 ts_tablet_manager.cc:1403] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.073s user 0.000s sys 0.004s
I20251024 19:48:27.998354 1766 raft_consensus.cc:359] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.998564 1766 raft_consensus.cc:385] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:27.998615 1766 raft_consensus.cc:740] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:27.998850 1766 consensus_queue.cc:260] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:27.999320 1766 ts_tablet_manager.cc:1434] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:27.999941 1766 tablet_bootstrap.cc:492] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:28.002789 1766 tablet_bootstrap.cc:654] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.020478 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_702f50f9fb0f4637b0b13e1457628397"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.021088 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_702f50f9fb0f4637b0b13e1457628397 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.025946 1184 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 46c76d86a68c40e18cc176b4776db266 (table default.Table_dbe9a03f95bf479490b2015a0c6afe20 [id=51716cd9fedf415bb03a962bb4235b42]). Expected version 1 got 0
I20251024 19:48:28.027691 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b32e44b4fac24495b6c62f393550ec11" } new_table_name: "default.Table_4229ef1c95b04ff384acc1ba3fd1d761" modify_external_catalogs: true
W20251024 19:48:28.029101 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ee9fb177b4a4dbc8fcd2c9f0b6ee699 consensus state has no leader
I20251024 19:48:28.029392 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4da9eaf31f3d4d80870061b13b95cad5" } modify_external_catalogs: true
I20251024 19:48:28.029448 1767 tablet_bootstrap.cc:492] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.029523 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4da9eaf31f3d4d80870061b13b95cad5" } modify_external_catalogs: true
I20251024 19:48:28.029616 1767 ts_tablet_manager.cc:1403] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.095s user 0.001s sys 0.024s
I20251024 19:48:28.030236 1192 catalog_manager.cc:5649] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43bec2f03fc849099e254f6335e9d532 (127.0.252.193). New cstate: current_term: 1 leader_uuid: "43bec2f03fc849099e254f6335e9d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:28.030395 1767 raft_consensus.cc:359] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.030575 1767 raft_consensus.cc:385] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.030591 1192 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet c1fc0f99ca5c42bfb9663e21a7b761c9 (table default.Table_d7da4325d747469b86ff66b85690867d [id=a32bd550682746f0a2861ff8be404992]). Expected version 1 got 0
I20251024 19:48:28.030627 1767 raft_consensus.cc:740] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.030826 1767 consensus_queue.cc:260] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.031343 1767 ts_tablet_manager.cc:1434] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251024 19:48:28.031632 1767 tablet_bootstrap.cc:492] T 292c31e7d4ce411bb336a1405ca43ba5 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:28.032147 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 350d9403cc9242839d7dca5ff60ae6b4 (DEFAULT_TABLE table=default.Table_1d23448dda3547c0a40cc1abbbd4ffc9 [id=fc7b87ea38574ef6a625cefb86c66d72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.032598 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 350d9403cc9242839d7dca5ff60ae6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.033221 1767 tablet_bootstrap.cc:654] T 292c31e7d4ce411bb336a1405ca43ba5 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.036249 3661 raft_consensus.cc:493] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.036401 3661 raft_consensus.cc:515] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.037040 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c54e569156042fca5e9ab0feaedf632 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:08 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.037082 3661 leader_election.cc:290] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.037678 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 15de0d8aa9e545c09233186bbe361b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.037937 1776 ts_tablet_manager.cc:1929] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251024 19:48:28.037961 1586 leader_election.cc:343] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: ab488e4971874a16b2e052c84c250777
I20251024 19:48:28.038051 1776 log.cc:1199] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/30bce965315d43948ecb8282c829c557
I20251024 19:48:28.038560 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab488e4971874a16b2e052c84c250777" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:28.038831 1776 ts_tablet_manager.cc:1950] T 30bce965315d43948ecb8282c829c557 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
W20251024 19:48:28.039199 1586 leader_election.cc:343] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.039304 1586 leader_election.cc:304] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:28.039294 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 350d9403cc9242839d7dca5ff60ae6b4 (DEFAULT_TABLE table=default.Table_1d23448dda3547c0a40cc1abbbd4ffc9 [id=fc7b87ea38574ef6a625cefb86c66d72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.039649 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 350d9403cc9242839d7dca5ff60ae6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.040087 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 350d9403cc9242839d7dca5ff60ae6b4 (DEFAULT_TABLE table=default.Table_1d23448dda3547c0a40cc1abbbd4ffc9 [id=fc7b87ea38574ef6a625cefb86c66d72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.041185 1337 tablet_service.cc:1505] Processing CreateTablet for tablet bf4174ffed9748268ab208b2d42978d8 (DEFAULT_TABLE table=default.Table_f041d5ea635b4c3db554dbcbf3b3c50f [id=a31cf63a866840fc95132d78faecef6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.041654 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf4174ffed9748268ab208b2d42978d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.041842 1775 ts_tablet_manager.cc:1929] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:28.041937 1775 log.cc:1199] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/769f576810ab49368eb0fc7837b7da58
I20251024 19:48:28.042286 1775 ts_tablet_manager.cc:1950] T 769f576810ab49368eb0fc7837b7da58 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:28.047231 1624 tablet_service.cc:1505] Processing CreateTablet for tablet bf4174ffed9748268ab208b2d42978d8 (DEFAULT_TABLE table=default.Table_f041d5ea635b4c3db554dbcbf3b3c50f [id=a31cf63a866840fc95132d78faecef6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.047704 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf4174ffed9748268ab208b2d42978d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.048329 1342 tablet_service.cc:1505] Processing CreateTablet for tablet d310b1fdc1eb4295a9f5b7a8ca4dfe6c (DEFAULT_TABLE table=default.Table_1f4e225434e5437e8119ca0959092f74 [id=3814b58966f74afe82b4f529bc4c6052]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.048746 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d310b1fdc1eb4295a9f5b7a8ca4dfe6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.050199 1629 tablet_service.cc:1505] Processing CreateTablet for tablet d310b1fdc1eb4295a9f5b7a8ca4dfe6c (DEFAULT_TABLE table=default.Table_1f4e225434e5437e8119ca0959092f74 [id=3814b58966f74afe82b4f529bc4c6052]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.050220 1485 tablet_service.cc:1505] Processing CreateTablet for tablet bf4174ffed9748268ab208b2d42978d8 (DEFAULT_TABLE table=default.Table_f041d5ea635b4c3db554dbcbf3b3c50f [id=a31cf63a866840fc95132d78faecef6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.050549 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d310b1fdc1eb4295a9f5b7a8ca4dfe6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.054206 3661 raft_consensus.cc:2749] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:28.059636 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 350d9403cc9242839d7dca5ff60ae6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.060235 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf4174ffed9748268ab208b2d42978d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.064096 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1253) took 1782 ms. Trace:
I20251024 19:48:28.064215 1776 rpcz_store.cc:276] 1024 19:48:26.281505 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.281798 (+ 293us) service_pool.cc:225] Handling call
1024 19:48:27.854796 (+1572998us) ts_tablet_manager.cc:1151] Deleting tablet 30bce965315d43948ecb8282c829c557
1024 19:48:27.854809 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.935539 (+ 80730us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.037828 (+102289us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.064070 (+ 26242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1572762}
I20251024 19:48:28.064704 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 30bce965315d43948ecb8282c829c557 (table default.Table_21c23f21b4f24c9989cdfe18a12a4693 [id=d9ae61ad6f8545d8a9945f08bdd731a3]) successfully deleted
I20251024 19:48:28.066794 1776 tablet_replica.cc:333] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:28.067106 1776 raft_consensus.cc:2243] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:28.067499 1776 raft_consensus.cc:2272] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:28.068339 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 121cfcb3a6dc4ed18484ac8eeabc13f7
I20251024 19:48:28.068351 1776 ts_tablet_manager.cc:1916] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:28.069137 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_a0344b12f5f74a59a78491ee2968442a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.069730 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0344b12f5f74a59a78491ee2968442a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.069770 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 350d9403cc9242839d7dca5ff60ae6b4
I20251024 19:48:28.070336 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6b2935707be843948a0ebf57ce97b78c" } modify_external_catalogs: true
I20251024 19:48:28.070482 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_6b2935707be843948a0ebf57ce97b78c" } modify_external_catalogs: true
I20251024 19:48:28.071348 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 19acf59898c8436e9d9fef98a7f07532
I20251024 19:48:28.071444 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 121cfcb3a6dc4ed18484ac8eeabc13f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.071496 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 404c38f8db2f43bfbc63e9ba2cee902c
I20251024 19:48:28.071666 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 350d9403cc9242839d7dca5ff60ae6b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.071736 1185 catalog_manager.cc:6719] ReplaceTablet: tablet 19acf59898c8436e9d9fef98a7f07532 deleted and replaced by tablet 9ffc598d59674ec2bcfcd8b656cc533f
I20251024 19:48:28.071825 1188 catalog_manager.cc:6719] ReplaceTablet: tablet 404c38f8db2f43bfbc63e9ba2cee902c deleted and replaced by tablet 40db8869778a46a0b8e26847d19753a6
I20251024 19:48:28.072086 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet f73069b9201449d08a4f7b246444e891 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:28.072175 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b7f521e0feb04a5fa8c1475090117699" } new_table_name: "default.Table_741666b9e2a84627b3e9714e3e89f3e9" modify_external_catalogs: true
I20251024 19:48:28.073482 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 6e419003a8644b0dbd4ce325a0b49d1e
I20251024 19:48:28.073690 1629 tablet_service.cc:1552] Processing DeleteTablet for tablet 404c38f8db2f43bfbc63e9ba2cee902c with delete_type TABLET_DATA_DELETED (replaced by tablet 40db8869778a46a0b8e26847d19753a6) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.073788 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 19acf59898c8436e9d9fef98a7f07532 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ffc598d59674ec2bcfcd8b656cc533f) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.074160 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_8927bbd785744aa199fa84cc4af4efce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.074643 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8927bbd785744aa199fa84cc4af4efce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.073491 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_db369541ede24bd69d04163bb5d48d62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.076056 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db369541ede24bd69d04163bb5d48d62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.077852 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e419003a8644b0dbd4ce325a0b49d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.079799 1766 tablet_bootstrap.cc:492] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:28.080063 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e419003a8644b0dbd4ce325a0b49d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.080140 1766 ts_tablet_manager.cc:1403] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.000s
I20251024 19:48:28.080238 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 19acf59898c8436e9d9fef98a7f07532 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ffc598d59674ec2bcfcd8b656cc533f) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.080389 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet 404c38f8db2f43bfbc63e9ba2cee902c with delete_type TABLET_DATA_DELETED (replaced by tablet 40db8869778a46a0b8e26847d19753a6) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.080063 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 121cfcb3a6dc4ed18484ac8eeabc13f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.080123 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 350d9403cc9242839d7dca5ff60ae6b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.081074 1766 raft_consensus.cc:359] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.081238 1766 raft_consensus.cc:385] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.081313 1766 raft_consensus.cc:740] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.081476 1766 consensus_queue.cc:260] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.082170 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 121cfcb3a6dc4ed18484ac8eeabc13f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.081243 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1134) took 4276 ms. Trace:
I20251024 19:48:28.082518 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 350d9403cc9242839d7dca5ff60ae6b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.082518 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e419003a8644b0dbd4ce325a0b49d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:27 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.082696 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 404c38f8db2f43bfbc63e9ba2cee902c with delete_type TABLET_DATA_DELETED (replaced by tablet 40db8869778a46a0b8e26847d19753a6) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.082696 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 19acf59898c8436e9d9fef98a7f07532 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ffc598d59674ec2bcfcd8b656cc533f) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.082373 1775 rpcz_store.cc:276] 1024 19:48:23.804442 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.804736 (+ 294us) service_pool.cc:225] Handling call
1024 19:48:27.969278 (+4164542us) ts_tablet_manager.cc:1151] Deleting tablet 769f576810ab49368eb0fc7837b7da58
1024 19:48:27.969291 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:27.984129 (+ 14838us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.041708 (+ 57579us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.081224 (+ 39516us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4164350}
I20251024 19:48:28.084045 1766 ts_tablet_manager.cc:1434] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:28.086163 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1135) took 4102 ms. Trace:
I20251024 19:48:28.086244 1775 rpcz_store.cc:276] 1024 19:48:23.983322 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:23.983375 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:28.083263 (+4099888us) ts_tablet_manager.cc:1151] Deleting tablet 03d8ab62b47f474198841ea14a3b7b00
1024 19:48:28.083561 (+ 298us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.086146 (+ 2585us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4099666}
W20251024 19:48:28.086452 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37de0de36bae403287da13caf065ed12 consensus state has no leader
I20251024 19:48:28.086712 1775 tablet_replica.cc:333] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:28.086850 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_0ef705da3bc84def887ca8ae8ee4a878" } new_table_name: "default.Table_e2fe881030b9494697581da39c43ebe2" modify_external_catalogs: true
I20251024 19:48:28.087466 1766 tablet_bootstrap.cc:492] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:28.088809 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 6e419003a8644b0dbd4ce325a0b49d1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e419003a8644b0dbd4ce325a0b49d1e already in progress: creating tablet
W20251024 19:48:28.089116 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 19acf59898c8436e9d9fef98a7f07532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19acf59898c8436e9d9fef98a7f07532 already in progress: creating tablet
I20251024 19:48:28.089258 1766 tablet_bootstrap.cc:654] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:28.089500 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 404c38f8db2f43bfbc63e9ba2cee902c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 404c38f8db2f43bfbc63e9ba2cee902c already in progress: creating tablet
I20251024 19:48:28.089720 1775 raft_consensus.cc:2243] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
W20251024 19:48:28.089874 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 121cfcb3a6dc4ed18484ac8eeabc13f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 121cfcb3a6dc4ed18484ac8eeabc13f7 already in progress: creating tablet
I20251024 19:48:28.089912 1775 raft_consensus.cc:2272] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
W20251024 19:48:28.090138 1179 catalog_manager.cc:4952] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 350d9403cc9242839d7dca5ff60ae6b4 because the tablet was not found. No further retry: Not found: Tablet not found: 350d9403cc9242839d7dca5ff60ae6b4
I20251024 19:48:28.090482 1775 ts_tablet_manager.cc:1916] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:28.092237 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 769f576810ab49368eb0fc7837b7da58 (table default.Table_2a9850f5d5dc499f87e87bdb0efcc93a [id=1da47e6010f548d1b3d8c538150a8a8e]) successfully deleted
W20251024 19:48:28.092521 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 03d8ab62b47f474198841ea14a3b7b00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03d8ab62b47f474198841ea14a3b7b00 already in progress: creating tablet
I20251024 19:48:28.097885 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 20e56d70049542f881f9490e4205dce3
I20251024 19:48:28.098693 3661 consensus_queue.cc:1048] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:28.098564 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_aadd52f1943441bda187b984246fcc02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.098878 1586 consensus_peers.cc:597] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee3c521876324808857f7724028c895f. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:28.099303 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aadd52f1943441bda187b984246fcc02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.099781 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 907d67bb08b94b1b979f2977240b0839 (DEFAULT_TABLE table=default.Table_c18a4e82a4db4df5a237651828a095c9 [id=6c5984b532ac4902ab7b5d4777ca9129]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.100222 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 907d67bb08b94b1b979f2977240b0839. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.100565 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 7142deef12dc4528b0cbb2b9fc8e8576 (DEFAULT_TABLE table=default.Table_12d28513e70345139f2f009dc4259026 [id=2f5ab2ea8c5d4d91964f1d6ad4d5f997]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.100898 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7142deef12dc4528b0cbb2b9fc8e8576. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.102499 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 20e56d70049542f881f9490e4205dce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.104593 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 4ff1159e3dd648b1a477ff08e60e287d (DEFAULT_TABLE table=default.Table_4ae2544614374b45bd7da7facda94f2c [id=4814ce48f4d14925a1d099c72f592b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.104912 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ff1159e3dd648b1a477ff08e60e287d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.099804 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 20e56d70049542f881f9490e4205dce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.108198 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet 20e56d70049542f881f9490e4205dce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.113025 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 907d67bb08b94b1b979f2977240b0839 (DEFAULT_TABLE table=default.Table_c18a4e82a4db4df5a237651828a095c9 [id=6c5984b532ac4902ab7b5d4777ca9129]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.113634 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 907d67bb08b94b1b979f2977240b0839. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.120302 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_e31cef9adc8a434eb92e694910da8a05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:28.120538 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_059135c028074f7fb4bc26da343a4de1" } new_table_name: "default.Table_04be5ba848b643b88315f422eeefba65" modify_external_catalogs: true
W20251024 19:48:28.120872 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e31cef9adc8a434eb92e694910da8a05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.124540 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 7142deef12dc4528b0cbb2b9fc8e8576 (DEFAULT_TABLE table=default.Table_12d28513e70345139f2f009dc4259026 [id=2f5ab2ea8c5d4d91964f1d6ad4d5f997]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.124990 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7142deef12dc4528b0cbb2b9fc8e8576. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.133168 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 4ff1159e3dd648b1a477ff08e60e287d (DEFAULT_TABLE table=default.Table_4ae2544614374b45bd7da7facda94f2c [id=4814ce48f4d14925a1d099c72f592b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.133510 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 73278db2097d4001a538d62dacb97c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:13 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.133730 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ff1159e3dd648b1a477ff08e60e287d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.134387 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 907d67bb08b94b1b979f2977240b0839 (DEFAULT_TABLE table=default.Table_c18a4e82a4db4df5a237651828a095c9 [id=6c5984b532ac4902ab7b5d4777ca9129]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.134790 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 907d67bb08b94b1b979f2977240b0839. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.136067 3607 raft_consensus.cc:493] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.136200 3607 raft_consensus.cc:515] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:28.136762 3607 leader_election.cc:290] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:28.137243 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b153f75caebb4d869e77529c9c9bb50d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:28.137411 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b153f75caebb4d869e77529c9c9bb50d" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:28.137805 1444 leader_election.cc:343] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:28.137926 1447 leader_election.cc:343] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.137986 1444 leader_election.cc:304] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:28.138476 3607 raft_consensus.cc:2749] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:28.138767 1339 tablet_service.cc:1552] Processing DeleteTablet for tablet 57f522a825264ea7b215304dbb7aa2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.139315 1774 tablet_replica.cc:333] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:28.139508 1774 raft_consensus.cc:2243] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:28.139657 1774 raft_consensus.cc:2272] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:28.140326 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.140389 1774 ts_tablet_manager.cc:1916] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:28.144515 1187 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 58a8f4f90f794e83a158e0026e5cea7f (table default.Table_c7b7dc69f2e640f6bb21208e8d8f32b8 [id=3fe81a9c4a134e908f4f2599c4b0cbf4]). Expected version 1 got 0
I20251024 19:48:28.146623 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a69c0918d2e4eed9d91b1913ec2c21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.146987 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet ab488e4971874a16b2e052c84c250777 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.147071 1774 ts_tablet_manager.cc:1929] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:28.147208 1774 log.cc:1199] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/57f522a825264ea7b215304dbb7aa2d5
I20251024 19:48:28.147616 1774 ts_tablet_manager.cc:1950] T 57f522a825264ea7b215304dbb7aa2d5 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:28.149144 1489 tablet_service.cc:1505] Processing CreateTablet for tablet d310b1fdc1eb4295a9f5b7a8ca4dfe6c (DEFAULT_TABLE table=default.Table_1f4e225434e5437e8119ca0959092f74 [id=3814b58966f74afe82b4f529bc4c6052]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.149139 1187 catalog_manager.cc:5649] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43bec2f03fc849099e254f6335e9d532 (127.0.252.193). New cstate: current_term: 1 leader_uuid: "43bec2f03fc849099e254f6335e9d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: HEALTHY } } }
I20251024 19:48:28.149807 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d310b1fdc1eb4295a9f5b7a8ca4dfe6c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:28.153541 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 20e56d70049542f881f9490e4205dce3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20e56d70049542f881f9490e4205dce3 already in progress: creating tablet
W20251024 19:48:28.154000 1179 catalog_manager.cc:4952] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 73278db2097d4001a538d62dacb97c9b because the tablet was not found. No further retry: Not found: Tablet not found: 73278db2097d4001a538d62dacb97c9b
I20251024 19:48:28.154209 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 57f522a825264ea7b215304dbb7aa2d5 (table default.Table_6a4dbd8e2b264bbca433946dfb1890b7 [id=9fec2fa31d474c5f8bc3b39041ed30bb]) successfully deleted
I20251024 19:48:28.149537 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 58a8f4f90f794e83a158e0026e5cea7f (table default.Table_c7b7dc69f2e640f6bb21208e8d8f32b8 [id=3fe81a9c4a134e908f4f2599c4b0cbf4]). Expected version 1 got 0
I20251024 19:48:28.155114 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 2b2992a738b440d798f5e61cc45ec088 (table default.Table_8523c29419f0445aa4da2ffdc6480f1a [id=40394401a8a84431bfb5e2b1db774832]). Expected version 4 got 0
I20251024 19:48:28.155375 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 4ff1159e3dd648b1a477ff08e60e287d (DEFAULT_TABLE table=default.Table_4ae2544614374b45bd7da7facda94f2c [id=4814ce48f4d14925a1d099c72f592b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.155874 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ff1159e3dd648b1a477ff08e60e287d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.157501 1191 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet b9ff74454a7e4803a7d0c307a21aef7a
I20251024 19:48:28.158979 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet f73069b9201449d08a4f7b246444e891
I20251024 19:48:28.159546 1185 catalog_manager.cc:6719] ReplaceTablet: tablet f73069b9201449d08a4f7b246444e891 deleted and replaced by tablet f5848ceed000455f96a199b5f9a70969
I20251024 19:48:28.160842 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet f73069b9201449d08a4f7b246444e891 with delete_type TABLET_DATA_DELETED (replaced by tablet f5848ceed000455f96a199b5f9a70969) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.166020 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 7142deef12dc4528b0cbb2b9fc8e8576 (DEFAULT_TABLE table=default.Table_12d28513e70345139f2f009dc4259026 [id=2f5ab2ea8c5d4d91964f1d6ad4d5f997]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:28.167116 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e96521c6871e45f08cd9db3190468f3e consensus state has no leader
I20251024 19:48:28.167097 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fb06db3f905c4de09f11a853224626b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.167848 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb06db3f905c4de09f11a853224626b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:28.169322 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5d0f73b12264834b0e7d3622f4ce706 has no consensus state
I20251024 19:48:28.169401 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fa228a9956134b5d823ae8abd11821a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:28.170157 1766 tablet_bootstrap.cc:492] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
W20251024 19:48:28.170243 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa228a9956134b5d823ae8abd11821a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.170295 1766 ts_tablet_manager.cc:1403] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.083s user 0.005s sys 0.000s
I20251024 19:48:28.171036 1766 raft_consensus.cc:359] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.171198 1766 raft_consensus.cc:385] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.171245 1766 raft_consensus.cc:740] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.176841 1776 ts_tablet_manager.cc:1929] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:28.179162 1776 log.cc:1199] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/c5cbce9e4b61451ba7f458df922be759
I20251024 19:48:28.179307 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7142deef12dc4528b0cbb2b9fc8e8576. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.180291 1776 ts_tablet_manager.cc:1950] T c5cbce9e4b61451ba7f458df922be759 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:28.183213 1766 consensus_queue.cc:260] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.184361 1775 ts_tablet_manager.cc:1929] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:28.185861 1775 log.cc:1199] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/80c58e2439dc490c824b5f0023b5434b
W20251024 19:48:28.186107 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74029a7d7cdc4503a833c1758887b684 consensus state has no leader
I20251024 19:48:28.186329 1775 ts_tablet_manager.cc:1950] T 80c58e2439dc490c824b5f0023b5434b P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:28.183898 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 830528cabe344ce89e5092dae613c7fe (DEFAULT_TABLE table=default.Table_a4cfda713b2348a3ae3ae5ca8454d76b [id=31acec0af7904ad2a128729671d232f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.187865 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 830528cabe344ce89e5092dae613c7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.189137 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet f73069b9201449d08a4f7b246444e891 with delete_type TABLET_DATA_DELETED (replaced by tablet f5848ceed000455f96a199b5f9a70969) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.189633 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet af43ffe097e14c4faffed95e207d0893 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.189651 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet f73069b9201449d08a4f7b246444e891 with delete_type TABLET_DATA_DELETED (replaced by tablet f5848ceed000455f96a199b5f9a70969) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.189893 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1136) took 4151 ms. Trace:
I20251024 19:48:28.190232 1775 rpcz_store.cc:276] 1024 19:48:24.038225 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.038296 (+ 71us) service_pool.cc:225] Handling call
1024 19:48:28.086604 (+4048308us) ts_tablet_manager.cc:1151] Deleting tablet 80c58e2439dc490c824b5f0023b5434b
1024 19:48:28.086676 (+ 72us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.132040 (+ 45364us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.184238 (+ 52198us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.189874 (+ 5636us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2769,"tablet-delete.queue_time_us":4048081}
I20251024 19:48:28.190949 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 7da1601a87524ef1a8b5069ba6823843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.191210 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 373aa543e1284fbc945ca802d9eddf59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.191406 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1137) took 4106 ms. Trace:
I20251024 19:48:28.191833 1768 tablet_bootstrap.cc:492] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:28.191766 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 9684072368f14cc2942b4e240f90b0f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.191949 1775 rpcz_store.cc:276] 1024 19:48:24.084749 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.084804 (+ 55us) service_pool.cc:225] Handling call
1024 19:48:28.191219 (+4106415us) ts_tablet_manager.cc:1151] Deleting tablet 0a5034103c1e4e039df7fc9f0ab65d0c
1024 19:48:28.191232 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.191389 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4106222}
I20251024 19:48:28.192257 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 7deb97be82964ea0b2fe9caa867a6460 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.192487 1768 ts_tablet_manager.cc:1403] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.203s user 0.004s sys 0.005s
I20251024 19:48:28.184556 1623 tablet_service.cc:1505] Processing CreateTablet for tablet c4b105534fd8493fb65775e2bd8d621b (DEFAULT_TABLE table=default.Table_845cc413de744780ad82fabf47646b46 [id=45a76c9a7e0b4942a29528efa3226ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.186693 1629 tablet_service.cc:1505] Processing CreateTablet for tablet ee45a6136789488d9a3e187dc991032b (DEFAULT_TABLE table=default.Table_323cd4e217904fa5a3671990e7b67ce9 [id=dce3d1f50b69460f848c66dde50df300]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.193409 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b105534fd8493fb65775e2bd8d621b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.193593 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee45a6136789488d9a3e187dc991032b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.193763 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1140) took 3935 ms. Trace:
I20251024 19:48:28.193835 1775 rpcz_store.cc:276] 1024 19:48:24.258394 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.258493 (+ 99us) service_pool.cc:225] Handling call
1024 19:48:28.192624 (+3934131us) ts_tablet_manager.cc:1151] Deleting tablet 33d42f4c818340cda4ae615132fcf03f
1024 19:48:28.192635 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.193745 (+ 1110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3930955}
I20251024 19:48:28.194074 1775 tablet_replica.cc:333] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:28.194083 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 830528cabe344ce89e5092dae613c7fe (DEFAULT_TABLE table=default.Table_a4cfda713b2348a3ae3ae5ca8454d76b [id=31acec0af7904ad2a128729671d232f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.194279 1775 raft_consensus.cc:2243] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:28.194432 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 830528cabe344ce89e5092dae613c7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.195369 1768 raft_consensus.cc:359] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.195542 1768 raft_consensus.cc:385] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.195586 1768 raft_consensus.cc:740] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.194432 1775 raft_consensus.cc:2272] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:28.196251 1775 ts_tablet_manager.cc:1916] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:28.197726 1768 consensus_queue.cc:260] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.200827 3607 raft_consensus.cc:493] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.200956 3607 raft_consensus.cc:515] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.193444 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 830528cabe344ce89e5092dae613c7fe (DEFAULT_TABLE table=default.Table_a4cfda713b2348a3ae3ae5ca8454d76b [id=31acec0af7904ad2a128729671d232f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.201669 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 830528cabe344ce89e5092dae613c7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.202224 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9684072368f14cc2942b4e240f90b0f2" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:28.202797 1444 leader_election.cc:343] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.203014 3607 leader_election.cc:290] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:28.203567 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9684072368f14cc2942b4e240f90b0f2" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:28.204072 1447 leader_election.cc:343] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.204226 1447 leader_election.cc:304] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:28.204654 3607 raft_consensus.cc:2749] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:28.205230 1768 ts_tablet_manager.cc:1434] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251024 19:48:28.205997 1768 tablet_bootstrap.cc:492] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:28.207571 1768 tablet_bootstrap.cc:654] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.211864 1339 tablet_service.cc:1505] Processing CreateTablet for tablet c4b105534fd8493fb65775e2bd8d621b (DEFAULT_TABLE table=default.Table_845cc413de744780ad82fabf47646b46 [id=45a76c9a7e0b4942a29528efa3226ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.212952 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b105534fd8493fb65775e2bd8d621b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.212467 1334 tablet_service.cc:1505] Processing CreateTablet for tablet ee45a6136789488d9a3e187dc991032b (DEFAULT_TABLE table=default.Table_323cd4e217904fa5a3671990e7b67ce9 [id=dce3d1f50b69460f848c66dde50df300]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.216959 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee45a6136789488d9a3e187dc991032b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.220373 1775 ts_tablet_manager.cc:1929] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:28.220988 1775 log.cc:1199] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/292c31e7d4ce411bb336a1405ca43ba5
I20251024 19:48:28.221527 1775 ts_tablet_manager.cc:1950] T 292c31e7d4ce411bb336a1405ca43ba5 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:28.222894 3599 raft_consensus.cc:493] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.223030 3599 raft_consensus.cc:515] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.223639 3599 leader_election.cc:290] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.224222 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1254) took 1902 ms. Trace:
I20251024 19:48:28.224357 1776 rpcz_store.cc:276] 1024 19:48:26.322177 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.322243 (+ 66us) service_pool.cc:225] Handling call
1024 19:48:28.066566 (+1744323us) ts_tablet_manager.cc:1151] Deleting tablet c5cbce9e4b61451ba7f458df922be759
1024 19:48:28.066698 (+ 132us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.137290 (+ 70592us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.176714 (+ 39424us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.224191 (+ 47477us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1744097}
I20251024 19:48:28.224956 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "786fe2fd6fca407e83a76f74f96a8b16" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:28.225030 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet c5cbce9e4b61451ba7f458df922be759 (table default.Table_f57b8c8910c2489a80979e2ffe453f3c [id=1817b735a20f46409680f85379280ca6]) successfully deleted
W20251024 19:48:28.225469 1305 leader_election.cc:343] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.226208 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "786fe2fd6fca407e83a76f74f96a8b16" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:28.226397 1508 raft_consensus.cc:2468] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:28.225531 1369 raft_consensus.cc:3060] T ccd46da86fe444baa675d3391cdd410c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:28.228317 1473 tablet_service.cc:1505] Processing CreateTablet for tablet c4b105534fd8493fb65775e2bd8d621b (DEFAULT_TABLE table=default.Table_845cc413de744780ad82fabf47646b46 [id=45a76c9a7e0b4942a29528efa3226ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.228681 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b105534fd8493fb65775e2bd8d621b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.230995 3639 raft_consensus.cc:493] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.231108 3639 raft_consensus.cc:515] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.231658 3639 leader_election.cc:290] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.234377 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b2992a738b440d798f5e61cc45ec088" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:28.234561 1508 raft_consensus.cc:2468] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:28.235018 1308 leader_election.cc:304] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:28.235337 3599 raft_consensus.cc:2804] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:28.235423 3599 raft_consensus.cc:493] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:28.235486 3599 raft_consensus.cc:3060] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:28.237351 1179 catalog_manager.cc:4952] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet c1cdd5ec32ca49c1bfafaf3a24050e89 because the tablet was not found. No further retry: Not found: Tablet not found: c1cdd5ec32ca49c1bfafaf3a24050e89
W20251024 19:48:28.237546 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet f73069b9201449d08a4f7b246444e891 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f73069b9201449d08a4f7b246444e891 already in progress: creating tablet
I20251024 19:48:28.238039 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 80c58e2439dc490c824b5f0023b5434b (table default.Table_2aa9405152c14d538c7164b50b336c7f [id=c00489e309ba4ada921f101a95bc8b48]) successfully deleted
W20251024 19:48:28.238242 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 0a5034103c1e4e039df7fc9f0ab65d0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a5034103c1e4e039df7fc9f0ab65d0c already in progress: creating tablet
W20251024 19:48:28.238440 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 33d42f4c818340cda4ae615132fcf03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33d42f4c818340cda4ae615132fcf03f already in progress: creating tablet
I20251024 19:48:28.238476 1308 leader_election.cc:304] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters:
I20251024 19:48:28.238837 3639 raft_consensus.cc:2804] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:28.238915 3639 raft_consensus.cc:493] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:28.238976 3639 raft_consensus.cc:3060] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:28.242959 3599 raft_consensus.cc:515] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.243822 3599 leader_election.cc:290] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.246575 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 8efe4911933745c68da1cf6a1b818a5f (DEFAULT_TABLE table=default.Table_4bd7d9d519c243d7acc4434dd39b18a0 [id=5518137c34254c17ab6e923bde31bec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.246776 1624 tablet_service.cc:1505] Processing CreateTablet for tablet dcf636eadc1640c281d21996ce4f3f56 (DEFAULT_TABLE table=default.Table_b7d69816c17a4e9cb68d17da3226cc02 [id=51c8dde91cd7425f95659899d62c8a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.247033 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8efe4911933745c68da1cf6a1b818a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.247205 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf636eadc1640c281d21996ce4f3f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.254762 3607 raft_consensus.cc:493] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.254923 3607 raft_consensus.cc:515] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.255517 3607 leader_election.cc:290] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:28.259626 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0d5d210d8d8499cbfdbdaa29e5bb40b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:28.260212 1444 leader_election.cc:343] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.246315 3639 raft_consensus.cc:515] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.260879 3607 raft_consensus.cc:493] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.261040 3607 raft_consensus.cc:515] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.261224 3639 leader_election.cc:290] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.261840 3607 leader_election.cc:290] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:28.263054 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af43ffe097e14c4faffed95e207d0893" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:28.263630 1444 leader_election.cc:343] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.264047 1766 ts_tablet_manager.cc:1434] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.094s user 0.000s sys 0.002s
I20251024 19:48:28.264587 1766 tablet_bootstrap.cc:492] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:28.265317 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b2992a738b440d798f5e61cc45ec088" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:28.266158 1766 tablet_bootstrap.cc:654] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.266238 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "786fe2fd6fca407e83a76f74f96a8b16" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:28.267069 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b2992a738b440d798f5e61cc45ec088" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:28.268994 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0d5d210d8d8499cbfdbdaa29e5bb40b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:28.269194 1367 raft_consensus.cc:2468] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:28.269766 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af43ffe097e14c4faffed95e207d0893" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:28.270310 1447 leader_election.cc:304] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:28.270619 3607 raft_consensus.cc:2804] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:28.270714 3607 raft_consensus.cc:493] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:28.270800 3607 raft_consensus.cc:3060] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:28.270998 1447 leader_election.cc:343] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.271111 1447 leader_election.cc:304] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:28.271459 3456 raft_consensus.cc:2749] T af43ffe097e14c4faffed95e207d0893 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:28.271778 1305 leader_election.cc:343] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:28.272086 1305 leader_election.cc:343] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:28.272243 1305 leader_election.cc:343] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.273312 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "786fe2fd6fca407e83a76f74f96a8b16" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:28.273499 1508 raft_consensus.cc:3060] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:28.274281 3607 raft_consensus.cc:515] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.275282 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0d5d210d8d8499cbfdbdaa29e5bb40b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
W20251024 19:48:28.275938 1444 leader_election.cc:343] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.276254 3607 leader_election.cc:290] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:28.276352 1508 raft_consensus.cc:2468] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:28.277367 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5fa8542957b14c0cb6e093a53d66eddd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:28.277410 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f4dae23457914d48b92be0b74929c89b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.278031 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fa8542957b14c0cb6e093a53d66eddd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.278630 1767 tablet_bootstrap.cc:492] T 292c31e7d4ce411bb336a1405ca43ba5 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.278786 1767 ts_tablet_manager.cc:1403] T 292c31e7d4ce411bb336a1405ca43ba5 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.247s user 0.015s sys 0.053s
I20251024 19:48:28.277426 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_04be5ba848b643b88315f422eeefba65" } new_table_name: "default.Table_b6e7436cebe44d00aa2a746d10100a1c" modify_external_catalogs: true
I20251024 19:48:28.279543 1767 raft_consensus.cc:359] T 292c31e7d4ce411bb336a1405ca43ba5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.279700 1767 raft_consensus.cc:385] T 292c31e7d4ce411bb336a1405ca43ba5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.279764 1767 raft_consensus.cc:740] T 292c31e7d4ce411bb336a1405ca43ba5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
W20251024 19:48:28.279804 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4dae23457914d48b92be0b74929c89b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.280035 1767 consensus_queue.cc:260] T 292c31e7d4ce411bb336a1405ca43ba5 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.280393 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1259) took 1776 ms. Trace:
I20251024 19:48:28.280503 1776 rpcz_store.cc:276] 1024 19:48:26.448011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.448071 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:28.224730 (+1776659us) ts_tablet_manager.cc:1151] Deleting tablet 1af368f347774270bca1a37c65d994ff
1024 19:48:28.224743 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.224869 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1776434}
I20251024 19:48:28.281597 1767 ts_tablet_manager.cc:1434] T 292c31e7d4ce411bb336a1405ca43ba5 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20251024 19:48:28.282001 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 1af368f347774270bca1a37c65d994ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1af368f347774270bca1a37c65d994ff already in progress: creating tablet
I20251024 19:48:28.282236 1767 tablet_bootstrap.cc:492] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:28.283550 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1260) took 1830 ms. Trace:
I20251024 19:48:28.283653 1776 rpcz_store.cc:276] 1024 19:48:26.452977 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.453044 (+ 67us) service_pool.cc:225] Handling call
1024 19:48:28.280865 (+1827821us) ts_tablet_manager.cc:1151] Deleting tablet 16a8c772ac404ae2b2f86d948a0fcd2f
1024 19:48:28.281295 (+ 430us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.283492 (+ 2197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1827617}
I20251024 19:48:28.283957 1776 tablet_replica.cc:333] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:28.283987 1473 tablet_service.cc:1505] Processing CreateTablet for tablet ee45a6136789488d9a3e187dc991032b (DEFAULT_TABLE table=default.Table_323cd4e217904fa5a3671990e7b67ce9 [id=dce3d1f50b69460f848c66dde50df300]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.284147 1776 raft_consensus.cc:2243] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:28.285872 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee45a6136789488d9a3e187dc991032b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.290591 1767 tablet_bootstrap.cc:654] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.297699 1767 tablet_bootstrap.cc:492] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.297910 1767 ts_tablet_manager.cc:1403] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20251024 19:48:28.297942 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 8efe4911933745c68da1cf6a1b818a5f (DEFAULT_TABLE table=default.Table_4bd7d9d519c243d7acc4434dd39b18a0 [id=5518137c34254c17ab6e923bde31bec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.298395 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8efe4911933745c68da1cf6a1b818a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.298785 1767 raft_consensus.cc:359] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.299036 1767 raft_consensus.cc:385] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.299135 1767 raft_consensus.cc:740] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.299379 1767 consensus_queue.cc:260] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:28.301023 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 16a8c772ac404ae2b2f86d948a0fcd2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16a8c772ac404ae2b2f86d948a0fcd2f already in progress: creating tablet
I20251024 19:48:28.301318 1624 tablet_service.cc:1505] Processing CreateTablet for tablet a403201fab014ea98734c3e80649e088 (DEFAULT_TABLE table=default.Table_de9a4e8e076649ce9e7aa790a4f1ba7f [id=877b65c08ba64fb798a9e137a17f4c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.301445 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 829b9cc3c19f479f80624deff66c6e5a (DEFAULT_TABLE table=default.Table_1f4d97aa9d894d43875a410601cc63b4 [id=b3e0c24d90c84b31b75ccc9da272e7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.301923 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 7ae56e17f6f64c55b190127a2aa7c450 (DEFAULT_TABLE table=default.Table_b693a6903f7e41da99547f6fb43144c0 [id=af65c940c0db432ea2ea37d5204f1c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.302350 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829b9cc3c19f479f80624deff66c6e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.302987 1612 tablet_service.cc:1505] Processing CreateTablet for tablet c75b9f86032e435e98f4623a0cc5edf0 (DEFAULT_TABLE table=default.Table_f97263739394418d85ec7e1bc0d54265 [id=cb29e599bf4445f595de64832b6b15f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.303431 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75b9f86032e435e98f4623a0cc5edf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.304853 1776 raft_consensus.cc:2272] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:28.305858 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ae56e17f6f64c55b190127a2aa7c450. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.309505 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a403201fab014ea98734c3e80649e088. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.313060 3607 raft_consensus.cc:493] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.313418 3607 raft_consensus.cc:515] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.313871 1766 tablet_bootstrap.cc:492] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:28.314023 1766 ts_tablet_manager.cc:1403] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20251024 19:48:28.314736 1766 raft_consensus.cc:359] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.314817 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c54e569156042fca5e9ab0feaedf632" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:28.314921 1766 raft_consensus.cc:385] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.314971 1766 raft_consensus.cc:740] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
W20251024 19:48:28.315394 1444 leader_election.cc:343] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.315884 1485 tablet_service.cc:1505] Processing CreateTablet for tablet a403201fab014ea98734c3e80649e088 (DEFAULT_TABLE table=default.Table_de9a4e8e076649ce9e7aa790a4f1ba7f [id=877b65c08ba64fb798a9e137a17f4c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.316287 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a403201fab014ea98734c3e80649e088. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.316239 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 829b9cc3c19f479f80624deff66c6e5a (DEFAULT_TABLE table=default.Table_1f4d97aa9d894d43875a410601cc63b4 [id=b3e0c24d90c84b31b75ccc9da272e7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.316468 1479 tablet_service.cc:1505] Processing CreateTablet for tablet dcf636eadc1640c281d21996ce4f3f56 (DEFAULT_TABLE table=default.Table_b7d69816c17a4e9cb68d17da3226cc02 [id=51c8dde91cd7425f95659899d62c8a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.316588 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829b9cc3c19f479f80624deff66c6e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.316782 1480 tablet_service.cc:1505] Processing CreateTablet for tablet c75b9f86032e435e98f4623a0cc5edf0 (DEFAULT_TABLE table=default.Table_f97263739394418d85ec7e1bc0d54265 [id=cb29e599bf4445f595de64832b6b15f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.317077 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 7ae56e17f6f64c55b190127a2aa7c450 (DEFAULT_TABLE table=default.Table_b693a6903f7e41da99547f6fb43144c0 [id=af65c940c0db432ea2ea37d5204f1c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.317288 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf636eadc1640c281d21996ce4f3f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.317487 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75b9f86032e435e98f4623a0cc5edf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.317456 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet d731403fbe0b4f92ac5899e13ea7bb19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.317898 1488 tablet_service.cc:1552] Processing DeleteTablet for tablet 7172a93a5bec4fa1b63282ba63beec0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:12 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.318102 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ae56e17f6f64c55b190127a2aa7c450. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.322515 1767 ts_tablet_manager.cc:1434] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20251024 19:48:28.322937 1767 tablet_bootstrap.cc:492] T c1fc0f99ca5c42bfb9663e21a7b761c9 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:28.324612 1767 tablet_bootstrap.cc:654] T c1fc0f99ca5c42bfb9663e21a7b761c9 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.324791 1766 consensus_queue.cc:260] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.327037 1766 ts_tablet_manager.cc:1434] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251024 19:48:28.332340 3599 raft_consensus.cc:493] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.332588 3599 raft_consensus.cc:515] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.333310 3599 leader_election.cc:290] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.335747 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b2992a738b440d798f5e61cc45ec088" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:28.335984 1508 raft_consensus.cc:3060] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:28.342459 3599 raft_consensus.cc:493] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.342890 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 8ec7897a92fe4698950c2f7533c559a1 (DEFAULT_TABLE table=default.Table_d6fedd4e6fbf4a718b3b5fd14d74f48b [id=7b49428dae10424db438ea9b0bc0d380]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.343348 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ec7897a92fe4698950c2f7533c559a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.343750 3607 leader_election.cc:290] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:28.344496 1586 consensus_peers.cc:597] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1cdd5ec32ca49c1bfafaf3a24050e89. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:28.344823 1586 consensus_peers.cc:597] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57f522a825264ea7b215304dbb7aa2d5. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:28.345359 1308 leader_election.cc:304] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
W20251024 19:48:28.345844 1308 leader_election.cc:343] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 292c31e7d4ce411bb336a1405ca43ba5
I20251024 19:48:28.346063 1766 tablet_bootstrap.cc:492] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:28.346184 3639 raft_consensus.cc:2804] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:28.342741 3599 raft_consensus.cc:515] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:28.348297 3599 leader_election.cc:290] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:28.350875 1766 tablet_bootstrap.cc:654] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.352126 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 37de0de36bae403287da13caf065ed12 (DEFAULT_TABLE table=default.Table_4229ef1c95b04ff384acc1ba3fd1d761 [id=c97160edf8354ab285e97b23fc4a1119]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.352165 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 786f0e890bb6410da9172eee32504be1 (DEFAULT_TABLE table=default.Table_ce86da7ef95441c88843d4f307fdb809 [id=13af13295fe740b991a636cfc088293f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.352572 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 786f0e890bb6410da9172eee32504be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.352494 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0d5d210d8d8499cbfdbdaa29e5bb40b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:28.352676 1370 raft_consensus.cc:3060] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:28.352807 1629 tablet_service.cc:1505] Processing CreateTablet for tablet b4f00b73001d434e8772a96c5de339c6 (DEFAULT_TABLE table=default.Table_12056492244041dabd5c64f523c27d7c [id=2035f1e7c6b44a658f8d30f62b62d8e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.352488 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292c31e7d4ce411bb336a1405ca43ba5" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:28.353060 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 6931d2a268354e5a9dad3bbc6e06989c (DEFAULT_TABLE table=default.Table_c86486fc320746d6a92ec7dc81c27180 [id=bcbe6237214c48ef9a6b086833373078]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.353163 1647 raft_consensus.cc:2468] T 292c31e7d4ce411bb336a1405ca43ba5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:28.353402 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "780442b310e6409498e6bbe4256280a4" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:28.355633 1370 raft_consensus.cc:2468] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:28.359759 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c54e569156042fca5e9ab0feaedf632" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:28.352807 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37de0de36bae403287da13caf065ed12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.361992 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6931d2a268354e5a9dad3bbc6e06989c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.362150 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4f00b73001d434e8772a96c5de339c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.362041 1334 tablet_service.cc:1505] Processing CreateTablet for tablet dcf636eadc1640c281d21996ce4f3f56 (DEFAULT_TABLE table=default.Table_b7d69816c17a4e9cb68d17da3226cc02 [id=51c8dde91cd7425f95659899d62c8a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.362757 1346 tablet_service.cc:1505] Processing CreateTablet for tablet a403201fab014ea98734c3e80649e088 (DEFAULT_TABLE table=default.Table_de9a4e8e076649ce9e7aa790a4f1ba7f [id=877b65c08ba64fb798a9e137a17f4c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.363333 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 8efe4911933745c68da1cf6a1b818a5f (DEFAULT_TABLE table=default.Table_4bd7d9d519c243d7acc4434dd39b18a0 [id=5518137c34254c17ab6e923bde31bec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.361685 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 786fe2fd6fca407e83a76f74f96a8b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.363823 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 780442b310e6409498e6bbe4256280a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.363910 1774 tablet_replica.cc:333] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:28.368888 1508 raft_consensus.cc:2468] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:28.371191 3573 raft_consensus.cc:493] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.371347 3573 raft_consensus.cc:515] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.372025 3573 leader_election.cc:290] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.372742 1503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83b91a5d764449d5b82ef6d1ee0360f5" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:28.372928 1503 raft_consensus.cc:2468] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:28.373324 1586 leader_election.cc:304] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:28.373759 3573 raft_consensus.cc:2804] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:28.373834 3573 raft_consensus.cc:493] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:28.373898 3573 raft_consensus.cc:3060] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:28.374787 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf636eadc1640c281d21996ce4f3f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.376530 3573 raft_consensus.cc:515] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.377103 3573 leader_election.cc:290] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.377710 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8efe4911933745c68da1cf6a1b818a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.378221 1503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83b91a5d764449d5b82ef6d1ee0360f5" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:28.378409 1503 raft_consensus.cc:3060] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:28.380558 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_702f50f9fb0f4637b0b13e1457628397" } modify_external_catalogs: true
I20251024 19:48:28.380770 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_702f50f9fb0f4637b0b13e1457628397" } modify_external_catalogs: true
I20251024 19:48:28.381395 1503 raft_consensus.cc:2468] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:28.382140 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_14a8b750c14d45f0af303e5b5202859e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:28.382362 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a403201fab014ea98734c3e80649e088. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:28.382781 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14a8b750c14d45f0af303e5b5202859e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.384693 1586 leader_election.cc:304] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:28.384999 3573 raft_consensus.cc:2804] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:28.385222 3573 raft_consensus.cc:697] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:28.385481 3573 consensus_queue.cc:237] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.385972 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_dab0576c96374dd0b1cd1409c1594039" } new_table_name: "default.Table_37671d233b204f0f81b63e1518018725" modify_external_catalogs: true
I20251024 19:48:28.387071 3607 raft_consensus.cc:493] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.387214 3607 raft_consensus.cc:515] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.388438 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d731403fbe0b4f92ac5899e13ea7bb19" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:28.389016 1444 leader_election.cc:343] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.389199 3639 raft_consensus.cc:697] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:28.389994 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3b4c85818a374983960d559f0e2612e5" } modify_external_catalogs: true
I20251024 19:48:28.389963 1191 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f145f64b26134e0e9fa7c293d5d7842f" } modify_external_catalogs: true
I20251024 19:48:28.390165 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3b4c85818a374983960d559f0e2612e5" } modify_external_catalogs: true
I20251024 19:48:28.390182 1191 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_f145f64b26134e0e9fa7c293d5d7842f" } modify_external_catalogs: true
I20251024 19:48:28.391469 3639 consensus_queue.cc:237] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.394670 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fe7a480771e74942ab56f7fdd69ecf23" } modify_external_catalogs: true
I20251024 19:48:28.394847 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_fe7a480771e74942ab56f7fdd69ecf23" } modify_external_catalogs: true
I20251024 19:48:28.395764 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 292c31e7d4ce411bb336a1405ca43ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.399588 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 292c31e7d4ce411bb336a1405ca43ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:28.401994 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74029a7d7cdc4503a833c1758887b684 consensus state has no leader
I20251024 19:48:28.406322 3607 leader_election.cc:290] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:28.408628 1776 ts_tablet_manager.cc:1916] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:28.410717 1308 leader_election.cc:304] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
W20251024 19:48:28.411055 1586 leader_election.cc:343] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 83b91a5d764449d5b82ef6d1ee0360f5
W20251024 19:48:28.411496 1586 leader_election.cc:343] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 83b91a5d764449d5b82ef6d1ee0360f5
I20251024 19:48:28.416000 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 8ec7897a92fe4698950c2f7533c559a1 (DEFAULT_TABLE table=default.Table_d6fedd4e6fbf4a718b3b5fd14d74f48b [id=7b49428dae10424db438ea9b0bc0d380]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.416716 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ec7897a92fe4698950c2f7533c559a1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:28.418653 1308 leader_election.cc:343] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 780442b310e6409498e6bbe4256280a4
I20251024 19:48:28.421742 3687 raft_consensus.cc:2804] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:28.427477 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 37de0de36bae403287da13caf065ed12 (DEFAULT_TABLE table=default.Table_4229ef1c95b04ff384acc1ba3fd1d761 [id=c97160edf8354ab285e97b23fc4a1119]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.429909 1485 tablet_service.cc:1505] Processing CreateTablet for tablet b4f00b73001d434e8772a96c5de339c6 (DEFAULT_TABLE table=default.Table_12056492244041dabd5c64f523c27d7c [id=2035f1e7c6b44a658f8d30f62b62d8e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.431592 1774 raft_consensus.cc:2243] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:28.432014 1774 pending_rounds.cc:70] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:28.432061 1774 pending_rounds.cc:77] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429423174815744 op_type: NO_OP noop_request { }
I20251024 19:48:28.432148 1774 raft_consensus.cc:2889] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:28.432200 1766 tablet_bootstrap.cc:492] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:28.432224 1774 raft_consensus.cc:2272] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:28.432329 1766 ts_tablet_manager.cc:1403] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.000s
I20251024 19:48:28.433046 1766 raft_consensus.cc:359] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:28.433218 1766 raft_consensus.cc:385] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.433277 1766 raft_consensus.cc:740] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.434031 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4f00b73001d434e8772a96c5de339c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.434244 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37de0de36bae403287da13caf065ed12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.434828 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 829b9cc3c19f479f80624deff66c6e5a (DEFAULT_TABLE table=default.Table_1f4d97aa9d894d43875a410601cc63b4 [id=b3e0c24d90c84b31b75ccc9da272e7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.435192 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829b9cc3c19f479f80624deff66c6e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.442044 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 7c690ea7e1bf4a6b8e2b33ce82159653 (DEFAULT_TABLE table=default.Table_2e8975d63d8c473785779f368a2d2601 [id=303358fd8cb64a5a96bc3b028a952f7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.442075 1339 tablet_service.cc:1505] Processing CreateTablet for tablet c75b9f86032e435e98f4623a0cc5edf0 (DEFAULT_TABLE table=default.Table_f97263739394418d85ec7e1bc0d54265 [id=cb29e599bf4445f595de64832b6b15f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.442452 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75b9f86032e435e98f4623a0cc5edf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.442736 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 786f0e890bb6410da9172eee32504be1 (DEFAULT_TABLE table=default.Table_ce86da7ef95441c88843d4f307fdb809 [id=13af13295fe740b991a636cfc088293f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.443032 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 786f0e890bb6410da9172eee32504be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.445894 1447 leader_election.cc:304] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532; no voters: af3952de050140d383f6b0de7666fa97
I20251024 19:48:28.446275 1305 leader_election.cc:304] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
W20251024 19:48:28.446341 1447 leader_election.cc:343] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.446435 1447 leader_election.cc:304] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:28.446662 1305 leader_election.cc:343] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.446763 1305 leader_election.cc:304] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:28.442413 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 7ae56e17f6f64c55b190127a2aa7c450 (DEFAULT_TABLE table=default.Table_b693a6903f7e41da99547f6fb43144c0 [id=af65c940c0db432ea2ea37d5204f1c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.447153 3679 raft_consensus.cc:2804] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:28.447266 3679 raft_consensus.cc:697] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:28.447463 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ae56e17f6f64c55b190127a2aa7c450. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.447535 3679 consensus_queue.cc:237] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.448568 3685 raft_consensus.cc:2749] T 4c54e569156042fca5e9ab0feaedf632 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:28.449318 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 37de0de36bae403287da13caf065ed12 (DEFAULT_TABLE table=default.Table_4229ef1c95b04ff384acc1ba3fd1d761 [id=c97160edf8354ab285e97b23fc4a1119]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.450191 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 8ec7897a92fe4698950c2f7533c559a1 (DEFAULT_TABLE table=default.Table_d6fedd4e6fbf4a718b3b5fd14d74f48b [id=7b49428dae10424db438ea9b0bc0d380]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.450611 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37de0de36bae403287da13caf065ed12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.454262 3599 raft_consensus.cc:2804] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:28.454388 3599 raft_consensus.cc:493] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:28.454509 3599 raft_consensus.cc:3060] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:28.456856 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ec7897a92fe4698950c2f7533c559a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.457118 3599 raft_consensus.cc:515] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.460987 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 786f0e890bb6410da9172eee32504be1 (DEFAULT_TABLE table=default.Table_ce86da7ef95441c88843d4f307fdb809 [id=13af13295fe740b991a636cfc088293f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.462057 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 786f0e890bb6410da9172eee32504be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.463651 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 6931d2a268354e5a9dad3bbc6e06989c (DEFAULT_TABLE table=default.Table_c86486fc320746d6a92ec7dc81c27180 [id=bcbe6237214c48ef9a6b086833373078]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.464078 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6931d2a268354e5a9dad3bbc6e06989c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.469830 1184 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 9d628eb57340459c973175b03d9deb44
I20251024 19:48:28.470216 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_153cb8026eb94425a5f0fc04046cd80a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.470806 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_153cb8026eb94425a5f0fc04046cd80a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.470868 1187 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 2b2992a738b440d798f5e61cc45ec088 (table default.Table_8523c29419f0445aa4da2ffdc6480f1a [id=40394401a8a84431bfb5e2b1db774832]). Expected version 4 got 0
I20251024 19:48:28.471895 1187 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_99ca9081a08f46faaaa0ad678def273a" } new_table_name: "default.Table_2f4fc376374e468d8d7d0f7a65f47f05" modify_external_catalogs: true
I20251024 19:48:28.471979 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_19a71f29de4849f5ad1d9e9741af2412"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.472389 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19a71f29de4849f5ad1d9e9741af2412 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.473299 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ab5fc4c134d94ab489257b3a84b1b485"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:28.473682 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4d52c098fe604c2f885442d39f4238b7" } modify_external_catalogs: true
I20251024 19:48:28.473847 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4d52c098fe604c2f885442d39f4238b7" } modify_external_catalogs: true
I20251024 19:48:28.476313 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0a49ed7967e3412d8bfd9c208fc8c67c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.476944 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a49ed7967e3412d8bfd9c208fc8c67c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:28.477865 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab5fc4c134d94ab489257b3a84b1b485 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:28.478499 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ffc598d59674ec2bcfcd8b656cc533f has no consensus state
I20251024 19:48:28.478902 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_d604f7e8552840359eae4a05ca6b65c5" } new_table_name: "default.Table_04e48889039f43688131634b37fc49e4" modify_external_catalogs: true
I20251024 19:48:28.481422 1766 consensus_queue.cc:260] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:28.482359 3404 raft_consensus.cc:493] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.482522 3404 raft_consensus.cc:515] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:28.482883 1612 tablet_service.cc:1505] Processing CreateTablet for tablet b94f7867bb9b4bbe96bf1bb6c2514eb3 (DEFAULT_TABLE table=default.Table_0b4fadd9a67b4f43965ba55cba4fe64e [id=f5e2be5fd0904d98bab73d75c79e9ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.483115 3404 leader_election.cc:290] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:28.483460 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 293ddf92a44b42a2a5356bbab9776709 (DEFAULT_TABLE table=default.Table_7ab1e070bda4405686f54e794245f399 [id=d4385a68af6f4930875385b855d17e36]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:28.484301 1586 leader_election.cc:343] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 80c58e2439dc490c824b5f0023b5434b
I20251024 19:48:28.485898 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c690ea7e1bf4a6b8e2b33ce82159653. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.486249 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 293ddf92a44b42a2a5356bbab9776709. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.486414 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b94f7867bb9b4bbe96bf1bb6c2514eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.493430 1766 ts_tablet_manager.cc:1434] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.061s user 0.002s sys 0.000s
I20251024 19:48:28.493793 1766 tablet_bootstrap.cc:492] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:28.495633 1766 tablet_bootstrap.cc:654] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.496407 1190 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 4d249f1216184b2a819a38245ff670ed
I20251024 19:48:28.497817 1767 tablet_bootstrap.cc:492] T c1fc0f99ca5c42bfb9663e21a7b761c9 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.498035 1767 ts_tablet_manager.cc:1403] T c1fc0f99ca5c42bfb9663e21a7b761c9 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.175s user 0.017s sys 0.019s
I20251024 19:48:28.498831 1767 raft_consensus.cc:359] T c1fc0f99ca5c42bfb9663e21a7b761c9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.499389 3599 leader_election.cc:290] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.499662 1767 raft_consensus.cc:385] T c1fc0f99ca5c42bfb9663e21a7b761c9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.499747 1767 raft_consensus.cc:740] T c1fc0f99ca5c42bfb9663e21a7b761c9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.500211 1767 consensus_queue.cc:260] T c1fc0f99ca5c42bfb9663e21a7b761c9 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.501641 1767 ts_tablet_manager.cc:1434] T c1fc0f99ca5c42bfb9663e21a7b761c9 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:28.502920 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1260f1d3582f4590983f08c4a7d0535f" } new_table_name: "default.Table_52aa7a7ba04d4184bc9c976c07387929" modify_external_catalogs: true
I20251024 19:48:28.501932 1767 tablet_bootstrap.cc:492] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:28.504726 1767 tablet_bootstrap.cc:654] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.510692 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 8f5fcc86923143a0903d02470ac06a56 (DEFAULT_TABLE table=default.Table_03461325877345a4aaa7e0437abd8813 [id=8dc2561a668e4cc1a94441cb7641ff21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.511359 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5fcc86923143a0903d02470ac06a56. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:28.513631 1586 leader_election.cc:343] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 80c58e2439dc490c824b5f0023b5434b
I20251024 19:48:28.513792 1586 leader_election.cc:304] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:28.514297 3573 raft_consensus.cc:2749] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:28.514818 1767 tablet_bootstrap.cc:492] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.514946 1767 ts_tablet_manager.cc:1403] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20251024 19:48:28.519338 1767 raft_consensus.cc:359] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:28.519526 1767 raft_consensus.cc:385] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.519598 1767 raft_consensus.cc:740] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.519824 1767 consensus_queue.cc:260] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:28.520295 1767 ts_tablet_manager.cc:1434] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251024 19:48:28.521683 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 7c690ea7e1bf4a6b8e2b33ce82159653 (DEFAULT_TABLE table=default.Table_2e8975d63d8c473785779f368a2d2601 [id=303358fd8cb64a5a96bc3b028a952f7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.522063 1767 tablet_bootstrap.cc:492] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:28.525398 1767 tablet_bootstrap.cc:654] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.528990 3694 consensus_queue.cc:1048] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:28.529376 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c690ea7e1bf4a6b8e2b33ce82159653. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:28.530540 1305 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.195:34119, user_credentials={real_user=slave}} blocked reactor thread for 83499.8us
I20251024 19:48:28.531590 1767 tablet_bootstrap.cc:492] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.531741 1767 ts_tablet_manager.cc:1403] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20251024 19:48:28.532025 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 6931d2a268354e5a9dad3bbc6e06989c (DEFAULT_TABLE table=default.Table_c86486fc320746d6a92ec7dc81c27180 [id=bcbe6237214c48ef9a6b086833373078]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.532299 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292c31e7d4ce411bb336a1405ca43ba5" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:28.532469 1648 raft_consensus.cc:3060] T 292c31e7d4ce411bb336a1405ca43ba5 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:28.532429 1767 raft_consensus.cc:359] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.532577 1767 raft_consensus.cc:385] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.532624 1767 raft_consensus.cc:740] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.532699 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6931d2a268354e5a9dad3bbc6e06989c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.532817 1767 consensus_queue.cc:260] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.533265 1767 ts_tablet_manager.cc:1434] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251024 19:48:28.533810 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 7c690ea7e1bf4a6b8e2b33ce82159653 (DEFAULT_TABLE table=default.Table_2e8975d63d8c473785779f368a2d2601 [id=303358fd8cb64a5a96bc3b028a952f7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.534327 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c690ea7e1bf4a6b8e2b33ce82159653. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.535233 1648 raft_consensus.cc:2468] T 292c31e7d4ce411bb336a1405ca43ba5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:28.535738 3679 raft_consensus.cc:493] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.536475 1767 tablet_bootstrap.cc:492] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:28.535874 3679 raft_consensus.cc:515] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.537271 3679 leader_election.cc:290] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:28.537920 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774a2de2e3764c54b72acfb952ed3e11" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:28.538118 1767 tablet_bootstrap.cc:654] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:28.538460 1444 leader_election.cc:343] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.539698 1485 tablet_service.cc:1505] Processing CreateTablet for tablet b94f7867bb9b4bbe96bf1bb6c2514eb3 (DEFAULT_TABLE table=default.Table_0b4fadd9a67b4f43965ba55cba4fe64e [id=f5e2be5fd0904d98bab73d75c79e9ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.540357 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b94f7867bb9b4bbe96bf1bb6c2514eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.540436 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 293ddf92a44b42a2a5356bbab9776709 (DEFAULT_TABLE table=default.Table_7ab1e070bda4405686f54e794245f399 [id=d4385a68af6f4930875385b855d17e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.540643 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d731403fbe0b4f92ac5899e13ea7bb19" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:28.540825 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 293ddf92a44b42a2a5356bbab9776709. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.460048 3692 raft_consensus.cc:2749] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:28.460208 3687 raft_consensus.cc:697] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:28.541262 3687 consensus_queue.cc:237] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.457538 1345 tablet_service.cc:1505] Processing CreateTablet for tablet b4f00b73001d434e8772a96c5de339c6 (DEFAULT_TABLE table=default.Table_12056492244041dabd5c64f523c27d7c [id=2035f1e7c6b44a658f8d30f62b62d8e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.542574 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4f00b73001d434e8772a96c5de339c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.542086 1767 tablet_bootstrap.cc:492] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.542754 1767 ts_tablet_manager.cc:1403] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20251024 19:48:28.543737 1767 raft_consensus.cc:359] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.543880 1767 raft_consensus.cc:385] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.543932 1767 raft_consensus.cc:740] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.545296 1767 consensus_queue.cc:260] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.546628 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 8f5fcc86923143a0903d02470ac06a56 (DEFAULT_TABLE table=default.Table_03461325877345a4aaa7e0437abd8813 [id=8dc2561a668e4cc1a94441cb7641ff21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.546753 1767 ts_tablet_manager.cc:1434] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:28.547062 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5fcc86923143a0903d02470ac06a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.547408 1767 tablet_bootstrap.cc:492] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:28.549535 1767 tablet_bootstrap.cc:654] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.553731 1774 ts_tablet_manager.cc:1916] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:28.556421 1767 tablet_bootstrap.cc:492] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.556556 1767 ts_tablet_manager.cc:1403] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20251024 19:48:28.557317 1767 raft_consensus.cc:359] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.557495 1767 raft_consensus.cc:385] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.559209 1767 raft_consensus.cc:740] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.559499 1767 consensus_queue.cc:260] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.560081 1767 ts_tablet_manager.cc:1434] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251024 19:48:28.560779 1774 ts_tablet_manager.cc:1929] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:28.560878 1774 log.cc:1199] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/786fe2fd6fca407e83a76f74f96a8b16
I20251024 19:48:28.561244 1774 ts_tablet_manager.cc:1950] T 786fe2fd6fca407e83a76f74f96a8b16 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:28.561734 1767 tablet_bootstrap.cc:492] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:28.562629 1776 ts_tablet_manager.cc:1929] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:28.562831 1776 log.cc:1199] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/73278db2097d4001a538d62dacb97c9b
I20251024 19:48:28.563395 1774 tablet_replica.cc:333] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:28.563635 1774 raft_consensus.cc:2243] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:28.563869 1774 raft_consensus.cc:2272] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:28.564455 1774 ts_tablet_manager.cc:1916] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:28.565145 1767 tablet_bootstrap.cc:654] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.571185 1767 tablet_bootstrap.cc:492] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.571337 1767 ts_tablet_manager.cc:1403] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20251024 19:48:28.572018 1767 raft_consensus.cc:359] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:28.572193 1767 raft_consensus.cc:385] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.572247 1767 raft_consensus.cc:740] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.572350 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1147) took 4184 ms. Trace:
I20251024 19:48:28.572482 1775 rpcz_store.cc:276] 1024 19:48:24.387687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.387852 (+ 165us) service_pool.cc:225] Handling call
1024 19:48:28.194037 (+3806185us) ts_tablet_manager.cc:1151] Deleting tablet 292c31e7d4ce411bb336a1405ca43ba5
1024 19:48:28.194045 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.210503 (+ 16458us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.220267 (+ 9764us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.572273 (+352006us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3763979}
I20251024 19:48:28.573035 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1172) took 3977 ms. Trace:
I20251024 19:48:28.575238 1767 consensus_queue.cc:260] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:28.575778 1586 consensus_peers.cc:597] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57f522a825264ea7b215304dbb7aa2d5. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:28.576993 1767 ts_tablet_manager.cc:1434] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251024 19:48:28.578389 1767 tablet_bootstrap.cc:492] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:48:28.581166 1308 leader_election.cc:343] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 292c31e7d4ce411bb336a1405ca43ba5
I20251024 19:48:28.582306 1767 tablet_bootstrap.cc:654] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.585094 1305 leader_election.cc:304] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
W20251024 19:48:28.584297 1447 leader_election.cc:343] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.588202 1447 leader_election.cc:304] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:28.588693 3679 raft_consensus.cc:2749] T d731403fbe0b4f92ac5899e13ea7bb19 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:28.588733 1767 tablet_bootstrap.cc:492] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.588864 1767 ts_tablet_manager.cc:1403] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20251024 19:48:28.586778 1766 tablet_bootstrap.cc:492] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:28.589087 1766 ts_tablet_manager.cc:1403] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.095s user 0.004s sys 0.000s
I20251024 19:48:28.589521 1767 raft_consensus.cc:359] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.589706 1767 raft_consensus.cc:385] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.589790 1767 raft_consensus.cc:740] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.590194 1766 raft_consensus.cc:359] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:28.590438 1766 raft_consensus.cc:385] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.590196 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 786fe2fd6fca407e83a76f74f96a8b16 (table default.Table_03318e0d18d34db48da05226fd40029a [id=f86c9d168b0e452899202f63a090f87a]) successfully deleted
I20251024 19:48:28.590539 1766 raft_consensus.cc:740] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.590852 1766 consensus_queue.cc:260] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:28.591113 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 292c31e7d4ce411bb336a1405ca43ba5 (table default.Table_2021df09852e4dd09dbf7b36a0a3ef5b [id=13f16de0ad1f43ac80639d3cf35fbb16]) successfully deleted
I20251024 19:48:28.591452 1766 ts_tablet_manager.cc:1434] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:28.591528 1767 consensus_queue.cc:260] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.593446 1767 ts_tablet_manager.cc:1434] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251024 19:48:28.594825 1767 tablet_bootstrap.cc:492] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:28.596971 1767 tablet_bootstrap.cc:654] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.592782 1775 rpcz_store.cc:276] 1024 19:48:24.595198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.595254 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:28.572904 (+3977650us) ts_tablet_manager.cc:1151] Deleting tablet 91bb076b4b0d4b04955043248460d04c
1024 19:48:28.572916 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.573020 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3977443}
I20251024 19:48:28.597548 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1173) took 3997 ms. Trace:
I20251024 19:48:28.597718 1775 rpcz_store.cc:276] 1024 19:48:24.599624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.599673 (+ 49us) service_pool.cc:225] Handling call
1024 19:48:28.597411 (+3997738us) ts_tablet_manager.cc:1151] Deleting tablet 9229fa6bf13441fa894e7bb7f78d7038
1024 19:48:28.597423 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.597528 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3997579}
I20251024 19:48:28.598462 1775 tablet_replica.cc:333] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
W20251024 19:48:28.600376 1444 consensus_peers.cc:597] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e -> Peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Couldn't send request to peer af3952de050140d383f6b0de7666fa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5cbce9e4b61451ba7f458df922be759. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:28.600440 1776 ts_tablet_manager.cc:1950] T 73278db2097d4001a538d62dacb97c9b P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:28.601028 1766 tablet_bootstrap.cc:492] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:28.602447 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1261) took 2149 ms. Trace:
I20251024 19:48:28.602648 1776 rpcz_store.cc:276] 1024 19:48:26.453121 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.453172 (+ 51us) service_pool.cc:225] Handling call
1024 19:48:28.283874 (+1830702us) ts_tablet_manager.cc:1151] Deleting tablet 73278db2097d4001a538d62dacb97c9b
1024 19:48:28.283884 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.496683 (+212799us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.562376 (+ 65693us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.602392 (+ 40016us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56449,"tablet-delete.queue_time_us":1830577}
I20251024 19:48:28.603021 1766 tablet_bootstrap.cc:654] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.603509 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 73278db2097d4001a538d62dacb97c9b (table default.Table_5114b358c69441539cfc49077ea63f9e [id=0568912ed6824b29bf68cd4f25ca2dff]) successfully deleted
I20251024 19:48:28.603617 1776 tablet_replica.cc:333] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:28.604018 1776 raft_consensus.cc:2243] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:28.604713 1776 pending_rounds.cc:70] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
W20251024 19:48:28.611719 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 91bb076b4b0d4b04955043248460d04c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91bb076b4b0d4b04955043248460d04c already in progress: creating tablet
W20251024 19:48:28.612720 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 9229fa6bf13441fa894e7bb7f78d7038 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9229fa6bf13441fa894e7bb7f78d7038 already in progress: creating tablet
I20251024 19:48:28.585222 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 8f5fcc86923143a0903d02470ac06a56 (DEFAULT_TABLE table=default.Table_03461325877345a4aaa7e0437abd8813 [id=8dc2561a668e4cc1a94441cb7641ff21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.614836 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5fcc86923143a0903d02470ac06a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.615482 3687 raft_consensus.cc:2804] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:28.615588 3687 raft_consensus.cc:697] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:28.615980 3687 consensus_queue.cc:237] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:28.584883 1342 consensus_meta.cc:287] T d310b1fdc1eb4295a9f5b7a8ca4dfe6c P 43bec2f03fc849099e254f6335e9d532: Time spent flushing consensus metadata: real 0.532s user 0.027s sys 0.000s
I20251024 19:48:28.585692 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774a2de2e3764c54b72acfb952ed3e11" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:28.586055 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 293ddf92a44b42a2a5356bbab9776709 (DEFAULT_TABLE table=default.Table_7ab1e070bda4405686f54e794245f399 [id=d4385a68af6f4930875385b855d17e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.626153 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 293ddf92a44b42a2a5356bbab9776709. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.630837 1768 tablet_bootstrap.cc:492] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:28.631181 1768 ts_tablet_manager.cc:1403] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.426s user 0.003s sys 0.007s
I20251024 19:48:28.632823 1768 raft_consensus.cc:359] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:28.633649 1768 raft_consensus.cc:385] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.633819 1768 raft_consensus.cc:740] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.636451 1191 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 3e09c12bcfd14ab081ab309f9dec4afc
W20251024 19:48:28.637810 1187 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b39e014607b4437b255c26c92c71861 consensus state has no leader
I20251024 19:48:28.638113 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2f47166e708545b68ae28fad4b98beb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:28.638460 1185 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet c1fc0f99ca5c42bfb9663e21a7b761c9 (table default.Table_d7da4325d747469b86ff66b85690867d [id=a32bd550682746f0a2861ff8be404992]). Expected version 1 got 0
W20251024 19:48:28.638631 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f47166e708545b68ae28fad4b98beb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.639240 1186 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 774a2de2e3764c54b72acfb952ed3e11 (table default.Table_5fb8e365cba24ded8061c38d9c695ff8 [id=1a8ca3395d7b471fa354154f2468080f]). Expected version 1 got 0
I20251024 19:48:28.637396 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c2674b5212a245e1933681c23c23e14b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:28.649650 1776 pending_rounds.cc:77] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429415595462656 op_type: NO_OP noop_request { }
I20251024 19:48:28.650177 1776 raft_consensus.cc:2889] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:28.650389 1776 raft_consensus.cc:2272] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:28.651130 1776 ts_tablet_manager.cc:1916] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:28.642849 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e09c12bcfd14ab081ab309f9dec4afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.659212 3573 raft_consensus.cc:493] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.659528 3573 raft_consensus.cc:515] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:28.661221 3573 leader_election.cc:290] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:28.639711 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c70bfd01ae774245b42810722e508b46 consensus state has no leader
I20251024 19:48:28.639495 1186 catalog_manager.cc:5649] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } }
I20251024 19:48:28.640090 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet bd2c43c6f497496d85cb35cc44f5ac8e
I20251024 19:48:28.598721 1775 raft_consensus.cc:2243] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:28.667281 1775 raft_consensus.cc:2272] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:28.653605 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet a27237314b684588aba9934065d9337b
I20251024 19:48:28.667960 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet bd2c43c6f497496d85cb35cc44f5ac8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.668566 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet a27237314b684588aba9934065d9337b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:28.640493 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2674b5212a245e1933681c23c23e14b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:28.671563 1586 leader_election.cc:343] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: 780442b310e6409498e6bbe4256280a4
W20251024 19:48:28.672497 1447 leader_election.cc:343] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.672617 1447 leader_election.cc:304] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:28.673373 3679 raft_consensus.cc:2749] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:28.678427 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet a27237314b684588aba9934065d9337b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.678678 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet bd2c43c6f497496d85cb35cc44f5ac8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.678979 1775 ts_tablet_manager.cc:1916] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:28.682193 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e09c12bcfd14ab081ab309f9dec4afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.682749 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet a27237314b684588aba9934065d9337b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.682749 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet bd2c43c6f497496d85cb35cc44f5ac8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:28.685664 1345 tablet_service.cc:1505] Processing CreateTablet for tablet b94f7867bb9b4bbe96bf1bb6c2514eb3 (DEFAULT_TABLE table=default.Table_0b4fadd9a67b4f43965ba55cba4fe64e [id=f5e2be5fd0904d98bab73d75c79e9ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:28.697734 3687 consensus_queue.cc:1048] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251024 19:48:28.706499 1648 raft_consensus.cc:3060] T c1fc0f99ca5c42bfb9663e21a7b761c9 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:28.709425 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e09c12bcfd14ab081ab309f9dec4afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.711148 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "780442b310e6409498e6bbe4256280a4" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:28.715448 1767 tablet_bootstrap.cc:492] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.715608 1767 ts_tablet_manager.cc:1403] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.121s user 0.013s sys 0.033s
I20251024 19:48:28.716328 1767 raft_consensus.cc:359] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.716507 1767 raft_consensus.cc:385] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.716565 1767 raft_consensus.cc:740] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.717443 1767 consensus_queue.cc:260] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.718669 1767 ts_tablet_manager.cc:1434] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:28.719211 1776 ts_tablet_manager.cc:1929] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:28.719307 1776 log.cc:1199] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/7be9290daabb404f955f05922d897fbf
I20251024 19:48:28.719403 1767 tablet_bootstrap.cc:492] T ebcc09d8e029451d8fb6b6601ad16087 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:28.719666 1776 ts_tablet_manager.cc:1950] T 7be9290daabb404f955f05922d897fbf P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:28.720808 1767 tablet_bootstrap.cc:654] T ebcc09d8e029451d8fb6b6601ad16087 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.725868 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1267) took 2214 ms. Trace:
I20251024 19:48:28.725971 1776 rpcz_store.cc:276] 1024 19:48:26.511114 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.513625 (+ 2511us) service_pool.cc:225] Handling call
1024 19:48:28.603535 (+2089910us) ts_tablet_manager.cc:1151] Deleting tablet 7be9290daabb404f955f05922d897fbf
1024 19:48:28.603546 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.654700 (+ 51154us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.719088 (+ 64388us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.725845 (+ 6757us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2089655}
I20251024 19:48:28.726346 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 7be9290daabb404f955f05922d897fbf (table default.Table_65097482f924409b87d7f4dcb5029849 [id=867c6e52ff2f4eb3a45495738770709d]) successfully deleted
W20251024 19:48:28.730193 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92d78642772d44179e67d75c51b9da4e consensus state has no leader
I20251024 19:48:28.732890 3686 raft_consensus.cc:493] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.733119 3686 raft_consensus.cc:515] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:28.734114 1184 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 0797d14e9fd846788164747523df0e2c (table default.Table_1f866642ce4a4ee885f169c4815bd1fb [id=a07755e138c84fd18c0fe72e81ab758b]). Expected version 1 got 0
I20251024 19:48:28.738163 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 786fe2fd6fca407e83a76f74f96a8b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:03 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.738294 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e6f8764e9ae472b9ac35a032182aed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.738440 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 780442b310e6409498e6bbe4256280a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.738869 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da1601a87524ef1a8b5069ba6823843" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:28.739540 1444 leader_election.cc:343] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.740566 1768 consensus_queue.cc:260] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:28.741398 1768 ts_tablet_manager.cc:1434] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.110s user 0.001s sys 0.001s
I20251024 19:48:28.744132 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet bc31e6c25ad14947af20abd766f12e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.744397 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 64499c8734d34c66b0b52a2cd3ef0673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.740229 3686 leader_election.cc:290] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:28.748265 3686 raft_consensus.cc:493] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.748520 3686 raft_consensus.cc:515] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.749501 1767 tablet_bootstrap.cc:492] T ebcc09d8e029451d8fb6b6601ad16087 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.749691 1767 ts_tablet_manager.cc:1403] T ebcc09d8e029451d8fb6b6601ad16087 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.009s
I20251024 19:48:28.749853 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1268) took 2238 ms. Trace:
I20251024 19:48:28.749939 1776 rpcz_store.cc:276] 1024 19:48:26.511189 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.513713 (+ 2524us) service_pool.cc:225] Handling call
1024 19:48:28.745923 (+2232210us) ts_tablet_manager.cc:1151] Deleting tablet 3cdd85a3ca7545eea953e149a6c0d572
1024 19:48:28.749719 (+ 3796us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.749834 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2232074}
I20251024 19:48:28.750239 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7172a93a5bec4fa1b63282ba63beec0c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:28.750370 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 3cdd85a3ca7545eea953e149a6c0d572 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cdd85a3ca7545eea953e149a6c0d572 already in progress: creating tablet
W20251024 19:48:28.750824 1444 leader_election.cc:343] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.751030 3686 leader_election.cc:290] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:28.752328 1776 tablet_replica.cc:333] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:28.752542 1776 raft_consensus.cc:2243] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:28.752882 1776 raft_consensus.cc:2272] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:28.752216 1767 raft_consensus.cc:359] T ebcc09d8e029451d8fb6b6601ad16087 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.753695 1767 raft_consensus.cc:385] T ebcc09d8e029451d8fb6b6601ad16087 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.754510 1767 raft_consensus.cc:740] T ebcc09d8e029451d8fb6b6601ad16087 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.755925 1776 ts_tablet_manager.cc:1916] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:28.757035 1767 consensus_queue.cc:260] T ebcc09d8e029451d8fb6b6601ad16087 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.766228 1767 ts_tablet_manager.cc:1434] T ebcc09d8e029451d8fb6b6601ad16087 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20251024 19:48:28.766942 1767 tablet_bootstrap.cc:492] T b22b58404b8b441ba97ef536885a66e0 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:28.769872 1766 tablet_bootstrap.cc:492] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:28.769898 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e427007b23f447b99ee7db28f8f4c085" } new_table_name: "default.Table_7f60bd2f528248ac94eff89a19d3de1b" modify_external_catalogs: true
I20251024 19:48:28.774618 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_37d5be312af34d6d920c4c7ad8acadc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.775435 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37d5be312af34d6d920c4c7ad8acadc5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.775738 1766 ts_tablet_manager.cc:1403] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.181s user 0.004s sys 0.000s
I20251024 19:48:28.775635 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3bca07553d454224b50597235f74a30a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.776909 1586 leader_election.cc:343] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: Tablet not RUNNING: STOPPED
I20251024 19:48:28.777042 1586 leader_election.cc:304] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
W20251024 19:48:28.777235 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bca07553d454224b50597235f74a30a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.777410 3573 raft_consensus.cc:2749] T 780442b310e6409498e6bbe4256280a4 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:28.782243 1766 raft_consensus.cc:359] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:28.782488 1766 raft_consensus.cc:385] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.782595 1766 raft_consensus.cc:740] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.782858 1766 consensus_queue.cc:260] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
W20251024 19:48:28.783712 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 424bce2f496b4a4fa74e57794f6a11eb consensus state has no leader
I20251024 19:48:28.790570 1184 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 58a8f4f90f794e83a158e0026e5cea7f (table default.Table_c7b7dc69f2e640f6bb21208e8d8f32b8 [id=3fe81a9c4a134e908f4f2599c4b0cbf4]). Expected version 1 got 0
I20251024 19:48:28.792292 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet ebcc09d8e029451d8fb6b6601ad16087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.799474 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3a2e0601bcf04513872c1b5ccd1b8bc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.800060 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a2e0601bcf04513872c1b5ccd1b8bc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.801600 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5fa8542957b14c0cb6e093a53d66eddd" } modify_external_catalogs: true
I20251024 19:48:28.801719 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_707ea75759974f808c67129a8651b7b5" } modify_external_catalogs: true
I20251024 19:48:28.801824 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5fa8542957b14c0cb6e093a53d66eddd" } modify_external_catalogs: true
I20251024 19:48:28.801885 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_707ea75759974f808c67129a8651b7b5" } modify_external_catalogs: true
I20251024 19:48:28.802877 1184 master_service.cc:954] ReplaceTablet: received request to replace tablet 20e56d70049542f881f9490e4205dce3 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:28.803377 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4ae2544614374b45bd7da7facda94f2c" } new_table_name: "default.Table_2c29884f33ad4581bb212e28ef728ab2" modify_external_catalogs: true
I20251024 19:48:28.768183 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da1601a87524ef1a8b5069ba6823843" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:28.773360 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7172a93a5bec4fa1b63282ba63beec0c" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:28.773845 3687 consensus_queue.cc:1048] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:28.805706 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_17b4b4daa67c4083845e023e3123c28e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.806427 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17b4b4daa67c4083845e023e3123c28e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.808024 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b94f7867bb9b4bbe96bf1bb6c2514eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:28.778625 1774 ts_tablet_manager.cc:1929] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:28.781250 3694 consensus_queue.cc:1048] T 58a8f4f90f794e83a158e0026e5cea7f P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:28.814464 3573 raft_consensus.cc:493] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.814646 3573 raft_consensus.cc:515] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.815264 3573 leader_election.cc:290] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.815989 1503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a69c0918d2e4eed9d91b1913ec2c21e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:28.816165 1503 raft_consensus.cc:2468] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:28.816543 1586 leader_election.cc:304] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:28.816838 3573 raft_consensus.cc:2804] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:28.816900 3573 raft_consensus.cc:493] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:28.816951 3573 raft_consensus.cc:3060] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:28.818943 1183 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet d5629d75793446bd9d0e970ba39c9357
I20251024 19:48:28.819621 3573 raft_consensus.cc:515] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.819891 3404 raft_consensus.cc:493] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.819995 3404 raft_consensus.cc:515] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.820514 3404 leader_election.cc:290] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.821156 1503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64499c8734d34c66b0b52a2cd3ef0673" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:28.821369 1503 raft_consensus.cc:2468] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:28.821961 1586 leader_election.cc:304] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:28.822204 3404 raft_consensus.cc:2804] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:28.822268 3404 raft_consensus.cc:493] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:28.822324 3404 raft_consensus.cc:3060] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:28.822343 1189 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ff1159e3dd648b1a477ff08e60e287d consensus state has no leader
I20251024 19:48:28.823858 1776 ts_tablet_manager.cc:1929] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:28.823989 1776 log.cc:1199] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/c1cdd5ec32ca49c1bfafaf3a24050e89
I20251024 19:48:28.824369 1776 ts_tablet_manager.cc:1950] T c1cdd5ec32ca49c1bfafaf3a24050e89 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:28.824739 1774 log.cc:1199] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/780442b310e6409498e6bbe4256280a4
I20251024 19:48:28.824963 1775 ts_tablet_manager.cc:1929] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:28.824954 3404 raft_consensus.cc:515] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.825043 1774 ts_tablet_manager.cc:1950] T 780442b310e6409498e6bbe4256280a4 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:28.825074 1775 log.cc:1199] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/83b91a5d764449d5b82ef6d1ee0360f5
I20251024 19:48:28.825328 1775 ts_tablet_manager.cc:1950] T 83b91a5d764449d5b82ef6d1ee0360f5 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:28.825487 3404 leader_election.cc:290] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.825531 1774 tablet_replica.cc:333] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:28.826058 1774 raft_consensus.cc:2243] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:28.826253 1503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64499c8734d34c66b0b52a2cd3ef0673" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:28.826413 1774 pending_rounds.cc:70] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:28.826408 1503 raft_consensus.cc:3060] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:28.826462 1774 pending_rounds.cc:77] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429424095436800 op_type: NO_OP noop_request { }
I20251024 19:48:28.826659 1774 raft_consensus.cc:2889] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:28.826758 1774 raft_consensus.cc:2272] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:28.827590 1767 tablet_bootstrap.cc:654] T b22b58404b8b441ba97ef536885a66e0 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.827893 1774 ts_tablet_manager.cc:1916] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:28.829103 1503 raft_consensus.cc:2468] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:28.829664 1586 leader_election.cc:304] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:28.829926 3661 raft_consensus.cc:2804] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:28.831612 3573 leader_election.cc:290] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.832360 1503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a69c0918d2e4eed9d91b1913ec2c21e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:28.833154 1503 raft_consensus.cc:3060] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:28.835227 1447 leader_election.cc:343] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.835355 1447 leader_election.cc:304] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:28.836496 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64499c8734d34c66b0b52a2cd3ef0673" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:28.836957 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64499c8734d34c66b0b52a2cd3ef0673" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:28.837240 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a69c0918d2e4eed9d91b1913ec2c21e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:28.837746 1586 leader_election.cc:343] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:28.838328 1586 leader_election.cc:343] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.839583 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a69c0918d2e4eed9d91b1913ec2c21e" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:28.839956 3686 raft_consensus.cc:2749] T 7172a93a5bec4fa1b63282ba63beec0c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:28.839965 1586 leader_election.cc:343] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:28.840430 1586 leader_election.cc:343] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.840770 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1276) took 2205 ms. Trace:
W20251024 19:48:28.841121 1447 leader_election.cc:343] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:28.841249 1447 leader_election.cc:304] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:28.843797 3686 raft_consensus.cc:2749] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:28.846006 1503 raft_consensus.cc:2468] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:28.848569 1768 tablet_bootstrap.cc:492] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:28.850518 1768 tablet_bootstrap.cc:654] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.846972 1586 leader_election.cc:304] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:28.854287 3573 raft_consensus.cc:2804] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:28.854434 3573 raft_consensus.cc:697] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:28.854740 3573 consensus_queue.cc:237] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:28.857939 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 74029a7d7cdc4503a833c1758887b684 (table default.Table_b6e7436cebe44d00aa2a746d10100a1c [id=d35a25cbeefd47d39a5cba0bed321da4], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 74029a7d7cdc4503a833c1758887b684 consensus state has no leader
I20251024 19:48:28.858006 1776 rpcz_store.cc:276] 1024 19:48:26.635127 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.638159 (+ 3032us) service_pool.cc:225] Handling call
1024 19:48:28.752283 (+2114124us) ts_tablet_manager.cc:1151] Deleting tablet c1cdd5ec32ca49c1bfafaf3a24050e89
1024 19:48:28.752292 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.759845 (+ 7553us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.823721 (+ 63876us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.840741 (+ 17020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2113789}
I20251024 19:48:28.858455 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1279) took 2198 ms. Trace:
I20251024 19:48:28.858525 1776 rpcz_store.cc:276] 1024 19:48:26.660155 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.660211 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:28.858314 (+2198103us) ts_tablet_manager.cc:1151] Deleting tablet d391bc3b3a144415bd4e6b5fdcc37612
1024 19:48:28.858324 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.858437 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2196504}
I20251024 19:48:28.858850 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1280) took 1774 ms. Trace:
I20251024 19:48:28.858929 1776 rpcz_store.cc:276] 1024 19:48:27.084397 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.084454 (+ 57us) service_pool.cc:225] Handling call
1024 19:48:28.858749 (+1774295us) ts_tablet_manager.cc:1151] Deleting tablet 60fbfb1620934034af2acaa39677554f
1024 19:48:28.858759 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.858832 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1774057}
I20251024 19:48:28.859099 3661 raft_consensus.cc:697] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:28.859128 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet c1cdd5ec32ca49c1bfafaf3a24050e89 (table default.Table_7d5114465a524d37b479ed5a6cebb938 [id=fab966f6dc2c4b299c517ea170531ab8]) successfully deleted
I20251024 19:48:28.859227 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1281) took 1768 ms. Trace:
I20251024 19:48:28.859285 1776 rpcz_store.cc:276] 1024 19:48:27.090887 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.092864 (+ 1977us) service_pool.cc:225] Handling call
1024 19:48:28.859132 (+1766268us) ts_tablet_manager.cc:1151] Deleting tablet ad21fd1f5d1f45fb99a0560e3405d69f
1024 19:48:28.859142 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.859213 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1766064}
I20251024 19:48:28.859362 3661 consensus_queue.cc:237] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.859566 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1282) took 1746 ms. Trace:
I20251024 19:48:28.859640 1776 rpcz_store.cc:276] 1024 19:48:27.112971 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.113030 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:28.859476 (+1746446us) ts_tablet_manager.cc:1151] Deleting tablet d933103af272429c95f8d3fc54e427da
1024 19:48:28.859485 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.859552 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1746262}
W20251024 19:48:28.859804 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d391bc3b3a144415bd4e6b5fdcc37612 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d391bc3b3a144415bd4e6b5fdcc37612 already in progress: creating tablet
I20251024 19:48:28.859949 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1283) took 1743 ms. Trace:
I20251024 19:48:28.860018 1776 rpcz_store.cc:276] 1024 19:48:27.116236 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.116287 (+ 51us) service_pool.cc:225] Handling call
1024 19:48:28.859856 (+1743569us) ts_tablet_manager.cc:1151] Deleting tablet 60f591124dbb444fa7a5b7f408d575df
1024 19:48:28.859866 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.859934 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1743401}
W20251024 19:48:28.860417 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 60fbfb1620934034af2acaa39677554f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60fbfb1620934034af2acaa39677554f already in progress: creating tablet
W20251024 19:48:28.860817 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet ad21fd1f5d1f45fb99a0560e3405d69f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad21fd1f5d1f45fb99a0560e3405d69f already in progress: creating tablet
W20251024 19:48:28.861179 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d933103af272429c95f8d3fc54e427da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d933103af272429c95f8d3fc54e427da already in progress: creating tablet
W20251024 19:48:28.861639 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 60f591124dbb444fa7a5b7f408d575df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60f591124dbb444fa7a5b7f408d575df already in progress: creating tablet
I20251024 19:48:28.864775 1767 tablet_bootstrap.cc:492] T b22b58404b8b441ba97ef536885a66e0 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.864945 1767 ts_tablet_manager.cc:1403] T b22b58404b8b441ba97ef536885a66e0 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.098s user 0.002s sys 0.002s
I20251024 19:48:28.865814 1767 raft_consensus.cc:359] T b22b58404b8b441ba97ef536885a66e0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.866039 1767 raft_consensus.cc:385] T b22b58404b8b441ba97ef536885a66e0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.866104 1767 raft_consensus.cc:740] T b22b58404b8b441ba97ef536885a66e0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.866348 1767 consensus_queue.cc:260] T b22b58404b8b441ba97ef536885a66e0 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:28.868088 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 65358.2us
I20251024 19:48:28.869132 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 780442b310e6409498e6bbe4256280a4 (table default.Table_7966ff9879d24254aed39562b2f127d7 [id=45b38d5e9fd6472fa318b41863080708]) successfully deleted
I20251024 19:48:28.869477 3738 raft_consensus.cc:493] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.869632 3738 raft_consensus.cc:515] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.870556 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8e8dbca79f4d72a2de51f105b209e4 with delete_type TABLET_DATA_DELETED (replaced by tablet fc50e9fbbefb43fda123ace9d1f32cf8) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:28.871090 1503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "786fe2fd6fca407e83a76f74f96a8b16" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:28.871287 1586 leader_election.cc:343] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 786fe2fd6fca407e83a76f74f96a8b16
I20251024 19:48:28.871408 1503 raft_consensus.cc:2393] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 43bec2f03fc849099e254f6335e9d532 in this term.
I20251024 19:48:28.871865 1586 leader_election.cc:304] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:28.872169 3404 raft_consensus.cc:3060] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:28.875454 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 157628us
I20251024 19:48:28.875598 3404 raft_consensus.cc:2749] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:28.876403 1766 ts_tablet_manager.cc:1434] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.095s user 0.001s sys 0.000s
I20251024 19:48:28.878386 3738 leader_election.cc:290] T 786fe2fd6fca407e83a76f74f96a8b16 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:28.883306 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1284) took 1765 ms. Trace:
I20251024 19:48:28.883498 1776 rpcz_store.cc:276] 1024 19:48:27.117995 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.118057 (+ 62us) service_pool.cc:225] Handling call
1024 19:48:28.860212 (+1742155us) ts_tablet_manager.cc:1151] Deleting tablet bd4deeaab67442898911dabfa7302492
1024 19:48:28.860222 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.883275 (+ 23053us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741995}
I20251024 19:48:28.885267 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1285) took 1649 ms. Trace:
W20251024 19:48:28.885200 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet bd4deeaab67442898911dabfa7302492 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4deeaab67442898911dabfa7302492 already in progress: creating tablet
I20251024 19:48:28.885921 1776 rpcz_store.cc:276] 1024 19:48:27.236247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.237651 (+ 1404us) service_pool.cc:225] Handling call
1024 19:48:28.885135 (+1647484us) ts_tablet_manager.cc:1151] Deleting tablet 1e45eefeeff949139f9407f9f90fba2a
1024 19:48:28.885147 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.885248 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1647259}
W20251024 19:48:28.886407 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 1e45eefeeff949139f9407f9f90fba2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e45eefeeff949139f9407f9f90fba2a already in progress: creating tablet
I20251024 19:48:28.886124 1767 ts_tablet_manager.cc:1434] T b22b58404b8b441ba97ef536885a66e0 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251024 19:48:28.887596 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1286) took 1629 ms. Trace:
I20251024 19:48:28.888100 1776 rpcz_store.cc:276] 1024 19:48:27.257788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.257970 (+ 182us) service_pool.cc:225] Handling call
1024 19:48:28.886411 (+1628441us) ts_tablet_manager.cc:1151] Deleting tablet e320bd0c6eba4d1b965e31d16234abcf
1024 19:48:28.886421 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.887573 (+ 1152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1628241}
I20251024 19:48:28.891395 1767 tablet_bootstrap.cc:492] T 92d78642772d44179e67d75c51b9da4e P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:28.900921 1767 tablet_bootstrap.cc:654] T 92d78642772d44179e67d75c51b9da4e P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.902477 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1174) took 4294 ms. Trace:
I20251024 19:48:28.902602 1775 rpcz_store.cc:276] 1024 19:48:24.608064 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.608659 (+ 595us) service_pool.cc:225] Handling call
1024 19:48:28.598414 (+3989755us) ts_tablet_manager.cc:1151] Deleting tablet 83b91a5d764449d5b82ef6d1ee0360f5
1024 19:48:28.598422 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.779484 (+181062us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.808119 (+ 28635us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.902449 (+ 94330us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3989556}
I20251024 19:48:28.903045 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1171) took 4311 ms. Trace:
I20251024 19:48:28.903115 1775 rpcz_store.cc:276] 1024 19:48:24.591875 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.634281 (+ 42406us) service_pool.cc:225] Handling call
1024 19:48:28.902918 (+4268637us) ts_tablet_manager.cc:1151] Deleting tablet 4b4904080b7a449fb92e773032808533
1024 19:48:28.902928 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.903029 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4268412}
W20251024 19:48:28.903903 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet e320bd0c6eba4d1b965e31d16234abcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e320bd0c6eba4d1b965e31d16234abcf already in progress: creating tablet
I20251024 19:48:28.904628 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1175) took 4260 ms. Trace:
I20251024 19:48:28.904760 1775 rpcz_store.cc:276] 1024 19:48:24.644132 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.655971 (+ 11839us) service_pool.cc:225] Handling call
1024 19:48:28.904498 (+4248527us) ts_tablet_manager.cc:1151] Deleting tablet 95e11ed28aab4e809e8fc1615f21744c
1024 19:48:28.904519 (+ 21us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.904611 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4243618}
I20251024 19:48:28.905341 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1182) took 4097 ms. Trace:
I20251024 19:48:28.905472 1775 rpcz_store.cc:276] 1024 19:48:24.807479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.817767 (+ 10288us) service_pool.cc:225] Handling call
1024 19:48:28.905210 (+4087443us) ts_tablet_manager.cc:1151] Deleting tablet 913fee0ca74e4cf99d6697ad5c38f4eb
1024 19:48:28.905223 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.905322 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4087073}
I20251024 19:48:28.905962 1775 tablet_replica.cc:333] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:28.906242 1775 raft_consensus.cc:2243] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:28.906637 1775 raft_consensus.cc:2272] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:28.907608 1775 ts_tablet_manager.cc:1916] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:28.913849 1767 tablet_bootstrap.cc:492] T 92d78642772d44179e67d75c51b9da4e P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:28.914244 1767 ts_tablet_manager.cc:1403] T 92d78642772d44179e67d75c51b9da4e P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.001s
I20251024 19:48:28.915850 1767 raft_consensus.cc:359] T 92d78642772d44179e67d75c51b9da4e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.916409 1767 raft_consensus.cc:385] T 92d78642772d44179e67d75c51b9da4e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.916514 1767 raft_consensus.cc:740] T 92d78642772d44179e67d75c51b9da4e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.917280 1767 consensus_queue.cc:260] T 92d78642772d44179e67d75c51b9da4e P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.918236 1767 ts_tablet_manager.cc:1434] T 92d78642772d44179e67d75c51b9da4e P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:28.922355 1766 tablet_bootstrap.cc:492] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:28.924196 1766 tablet_bootstrap.cc:654] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.936800 1775 ts_tablet_manager.cc:1929] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:28.938608 3661 raft_consensus.cc:493] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:28.938781 3661 raft_consensus.cc:515] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.939059 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 83b91a5d764449d5b82ef6d1ee0360f5 (table default.Table_325e96b0d32e4a3aab388a916c29ec26 [id=c6b5c56913684d08b89fde458e31b8cb]) successfully deleted
W20251024 19:48:28.939294 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 4b4904080b7a449fb92e773032808533 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b4904080b7a449fb92e773032808533 already in progress: creating tablet
I20251024 19:48:28.939411 3661 leader_election.cc:290] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:28.939604 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 913fee0ca74e4cf99d6697ad5c38f4eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 913fee0ca74e4cf99d6697ad5c38f4eb already in progress: creating tablet
I20251024 19:48:28.940255 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0d5d210d8d8499cbfdbdaa29e5bb40b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:28.940480 1369 raft_consensus.cc:2393] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
I20251024 19:48:28.940482 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f0f1e12cae86473ea751143cac835f76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:28.940603 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6b5b031c418245ab8b67ee0e08dccfcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:28.941072 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b5b031c418245ab8b67ee0e08dccfcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:28.941077 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0f1e12cae86473ea751143cac835f76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.954905 1768 tablet_bootstrap.cc:492] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:28.955169 1768 ts_tablet_manager.cc:1403] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.151s user 0.003s sys 0.000s
I20251024 19:48:28.956002 1768 raft_consensus.cc:359] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.956386 1768 raft_consensus.cc:385] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:28.956496 1768 raft_consensus.cc:740] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:28.956806 1768 consensus_queue.cc:260] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:28.960762 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1287) took 1681 ms. Trace:
I20251024 19:48:28.960860 1776 rpcz_store.cc:276] 1024 19:48:27.279655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.279725 (+ 70us) service_pool.cc:225] Handling call
1024 19:48:28.901905 (+1622180us) ts_tablet_manager.cc:1151] Deleting tablet 3e65583d0ea84a3ea10ae11462eac663
1024 19:48:28.960592 (+ 58687us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.960723 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1621999}
I20251024 19:48:28.961140 1776 tablet_replica.cc:333] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:28.961352 1776 raft_consensus.cc:2243] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:28.961653 1776 pending_rounds.cc:70] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:28.961704 1776 pending_rounds.cc:77] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429417574526976 op_type: NO_OP noop_request { }
I20251024 19:48:28.961803 1776 raft_consensus.cc:2889] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:28.961884 1776 raft_consensus.cc:2272] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:28.962441 1776 ts_tablet_manager.cc:1916] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:28.964170 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 58a8f4f90f794e83a158e0026e5cea7f (table default.Table_c7b7dc69f2e640f6bb21208e8d8f32b8 [id=3fe81a9c4a134e908f4f2599c4b0cbf4]). Expected version 1 got 0
I20251024 19:48:28.964401 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet c1fc0f99ca5c42bfb9663e21a7b761c9 (table default.Table_d7da4325d747469b86ff66b85690867d [id=a32bd550682746f0a2861ff8be404992]). Expected version 1 got 0
I20251024 19:48:28.966243 1767 tablet_bootstrap.cc:492] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:28.975236 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet b22b58404b8b441ba97ef536885a66e0 with delete_type TABLET_DATA_DELETED (replaced by tablet f37a01dc3d5f458d81605f2c41fa27dd) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:28.978932 1767 tablet_bootstrap.cc:654] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:28.985597 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a4cfda713b2348a3ae3ae5ca8454d76b" } modify_external_catalogs: true
I20251024 19:48:28.985782 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a4cfda713b2348a3ae3ae5ca8454d76b" } modify_external_catalogs: true
I20251024 19:48:28.986726 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2ff196bb7bb24e7e9bdecaa140baf000"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:28.986944 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5fe87e2cf20f469689cec015ffe87708" } modify_external_catalogs: true
I20251024 19:48:28.987149 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5fe87e2cf20f469689cec015ffe87708" } modify_external_catalogs: true
W20251024 19:48:28.987326 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ff196bb7bb24e7e9bdecaa140baf000 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:28.987954 1185 master_service.cc:954] ReplaceTablet: received request to replace tablet 4c54e569156042fca5e9ab0feaedf632 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:28.988821 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_21c23f21b4f24c9989cdfe18a12a4693" } new_table_name: "default.Table_82ab16da2ef848e098a10530d7d6f215" modify_external_catalogs: true
I20251024 19:48:28.990312 1188 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_bf15d3c700774490b1860d3d8865a95e" } modify_external_catalogs: true
I20251024 19:48:28.990494 1188 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_bf15d3c700774490b1860d3d8865a95e" } modify_external_catalogs: true
I20251024 19:48:28.991714 1776 ts_tablet_manager.cc:1929] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:28.991885 1776 log.cc:1199] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/fc7ccf1495924782b9b6ac1173bc73a1
I20251024 19:48:28.992051 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_539d0fd237684238a18e3abd9599b632" } new_table_name: "default.Table_c3df6b8ddfaa44c68342d0eafc0f7e79" modify_external_catalogs: true
I20251024 19:48:28.992245 1776 ts_tablet_manager.cc:1950] T fc7ccf1495924782b9b6ac1173bc73a1 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:28.994451 1768 ts_tablet_manager.cc:1434] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20251024 19:48:28.940822 1506 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0d5d210d8d8499cbfdbdaa29e5bb40b" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:28.993992 1766 tablet_bootstrap.cc:492] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:28.997749 1766 ts_tablet_manager.cc:1403] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.075s user 0.004s sys 0.000s
I20251024 19:48:28.998296 1775 log.cc:1199] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/c1cdd5ec32ca49c1bfafaf3a24050e89
I20251024 19:48:29.001960 1775 ts_tablet_manager.cc:1950] T c1cdd5ec32ca49c1bfafaf3a24050e89 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:29.002153 1768 tablet_bootstrap.cc:492] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:28.993099 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1288) took 1639 ms. Trace:
I20251024 19:48:29.008272 1776 rpcz_store.cc:276] 1024 19:48:27.354085 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.354150 (+ 65us) service_pool.cc:225] Handling call
1024 19:48:28.961102 (+1606952us) ts_tablet_manager.cc:1151] Deleting tablet fc7ccf1495924782b9b6ac1173bc73a1
1024 19:48:28.961110 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.984100 (+ 22990us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.989470 (+ 5370us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.993071 (+ 3601us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1606733}
I20251024 19:48:28.997534 1586 leader_election.cc:304] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.009308 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet fc7ccf1495924782b9b6ac1173bc73a1 (table default.Table_6a5150d7236d4d558c9f9fb4e70a78dd [id=991b0e462eba40ec882e422e60b43f26]) successfully deleted
I20251024 19:48:29.009509 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1291) took 1655 ms. Trace:
I20251024 19:48:29.009558 3661 raft_consensus.cc:3060] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.012987 3661 raft_consensus.cc:2749] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:29.013661 1767 tablet_bootstrap.cc:492] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:29.013841 1767 ts_tablet_manager.cc:1403] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.000s
I20251024 19:48:29.014917 1767 raft_consensus.cc:359] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.015102 1767 raft_consensus.cc:385] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.015153 1767 raft_consensus.cc:740] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.015359 1767 consensus_queue.cc:260] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.016147 1767 ts_tablet_manager.cc:1434] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:29.017544 1767 tablet_bootstrap.cc:492] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:29.020543 3686 raft_consensus.cc:493] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.020689 3686 raft_consensus.cc:515] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.021345 3686 leader_election.cc:290] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:29.022042 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1295c99836d14a1cb0002e6c6aaee1eb" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:29.022367 1188 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2d01572e51694d9bb58c3a59d4205102
W20251024 19:48:29.022655 1444 leader_election.cc:343] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.023298 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d01572e51694d9bb58c3a59d4205102 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.024816 1767 tablet_bootstrap.cc:654] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.025827 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1295c99836d14a1cb0002e6c6aaee1eb" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:29.026321 1447 leader_election.cc:343] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.026437 1447 leader_election.cc:304] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:29.026929 3686 raft_consensus.cc:2749] T 1295c99836d14a1cb0002e6c6aaee1eb P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:29.027973 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 157f84b56067415da404df30fc8a5755 consensus state has no leader
I20251024 19:48:29.042196 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d01572e51694d9bb58c3a59d4205102 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:29.043843 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d01572e51694d9bb58c3a59d4205102 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.007730 1766 raft_consensus.cc:359] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.044082 1766 raft_consensus.cc:385] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.044170 1766 raft_consensus.cc:740] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.044456 1766 consensus_queue.cc:260] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.046033 1766 ts_tablet_manager.cc:1434] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20251024 19:48:29.046221 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1190) took 4129 ms. Trace:
I20251024 19:48:29.046281 1775 rpcz_store.cc:276] 1024 19:48:24.916997 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:24.917975 (+ 978us) service_pool.cc:225] Handling call
1024 19:48:28.905917 (+3987942us) ts_tablet_manager.cc:1151] Deleting tablet c1cdd5ec32ca49c1bfafaf3a24050e89
1024 19:48:28.905926 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:28.933417 (+ 27491us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:28.936704 (+ 3287us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:29.046204 (+109500us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3987775}
I20251024 19:48:29.009619 1776 rpcz_store.cc:276] 1024 19:48:27.354402 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.354453 (+ 51us) service_pool.cc:225] Handling call
1024 19:48:29.009393 (+1654940us) ts_tablet_manager.cc:1151] Deleting tablet d1d26e02ad2945b39cf27738ed5b7cf4
1024 19:48:29.009405 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.009489 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654789}
I20251024 19:48:29.047039 1776 tablet_replica.cc:333] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:29.047211 1776 raft_consensus.cc:2243] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:29.047554 1776 pending_rounds.cc:70] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:29.047602 1776 pending_rounds.cc:77] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429417420333056 op_type: NO_OP noop_request { }
I20251024 19:48:29.047705 1776 raft_consensus.cc:2889] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20251024 19:48:29.047706 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet d1d26e02ad2945b39cf27738ed5b7cf4 because the tablet was not found. No further retry: Not found: Tablet not found: d1d26e02ad2945b39cf27738ed5b7cf4
I20251024 19:48:29.047775 1776 raft_consensus.cc:2272] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:29.048344 1776 ts_tablet_manager.cc:1916] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:29.048943 3639 raft_consensus.cc:493] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.049142 3639 raft_consensus.cc:515] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.050238 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab488e4971874a16b2e052c84c250777" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:29.050297 3639 leader_election.cc:290] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.050419 1648 raft_consensus.cc:2468] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 0.
I20251024 19:48:29.050796 1305 leader_election.cc:304] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:29.051204 3639 raft_consensus.cc:2804] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:29.051318 3639 raft_consensus.cc:493] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:29.051424 3639 raft_consensus.cc:3060] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:29.052045 1308 leader_election.cc:343] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: ab488e4971874a16b2e052c84c250777
I20251024 19:48:29.054492 3639 raft_consensus.cc:515] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.055505 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab488e4971874a16b2e052c84c250777" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:29.055662 1648 raft_consensus.cc:3060] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:29.056675 1308 leader_election.cc:343] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: ab488e4971874a16b2e052c84c250777
I20251024 19:48:29.057365 1768 tablet_bootstrap.cc:654] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.058452 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1197) took 4023 ms. Trace:
I20251024 19:48:29.058487 1648 raft_consensus.cc:2468] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43bec2f03fc849099e254f6335e9d532 in term 1.
I20251024 19:48:29.059084 1305 leader_election.cc:304] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e
I20251024 19:48:29.059387 3694 raft_consensus.cc:2804] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:29.059492 3694 raft_consensus.cc:697] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Becoming Leader. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Running, Role: LEADER
I20251024 19:48:29.059769 3694 consensus_queue.cc:237] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.068030 1767 tablet_bootstrap.cc:492] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:29.068504 1767 ts_tablet_manager.cc:1403] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.002s
I20251024 19:48:29.069425 1767 raft_consensus.cc:359] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.069651 1767 raft_consensus.cc:385] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.069720 1767 raft_consensus.cc:740] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.069965 1767 consensus_queue.cc:260] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.070467 3639 leader_election.cc:290] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.070806 1767 ts_tablet_manager.cc:1434] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:29.064587 1766 tablet_bootstrap.cc:492] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:29.072302 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet c1cdd5ec32ca49c1bfafaf3a24050e89 (table default.Table_7d5114465a524d37b479ed5a6cebb938 [id=fab966f6dc2c4b299c517ea170531ab8]) successfully deleted
I20251024 19:48:29.070659 1775 rpcz_store.cc:276] 1024 19:48:25.035461 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.036192 (+ 731us) service_pool.cc:225] Handling call
1024 19:48:29.046657 (+4010465us) ts_tablet_manager.cc:1151] Deleting tablet 24ff2839ff1e48d5a2f84890ec25c75b
1024 19:48:29.058300 (+ 11643us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.058426 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4010303}
I20251024 19:48:29.073369 1767 tablet_bootstrap.cc:492] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:29.072999 1766 tablet_bootstrap.cc:654] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.076758 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1198) took 3998 ms. Trace:
I20251024 19:48:29.077608 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_22c143aabac54b76afbf3cf41fac7d57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:29.077836 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f331734f70c84eb8aeaf149f4168a2fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.078151 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22c143aabac54b76afbf3cf41fac7d57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:29.078356 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f331734f70c84eb8aeaf149f4168a2fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.079456 1767 tablet_bootstrap.cc:654] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.088318 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e31cef9adc8a434eb92e694910da8a05" } new_table_name: "default.Table_ae482796a941475d8e4597fa4a8a1264" modify_external_catalogs: true
I20251024 19:48:29.083928 1775 rpcz_store.cc:276] 1024 19:48:25.077881 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.078140 (+ 259us) service_pool.cc:225] Handling call
1024 19:48:29.076539 (+3998399us) ts_tablet_manager.cc:1151] Deleting tablet 018d3e6aba6740f0bd82e8c83863bc4e
1024 19:48:29.076555 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.076734 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3996717}
I20251024 19:48:29.092666 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7c327a6167cf43449deb0db222d887ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:29.092885 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1200) took 4014 ms. Trace:
I20251024 19:48:29.092985 1775 rpcz_store.cc:276] 1024 19:48:25.078074 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.079221 (+ 1147us) service_pool.cc:225] Handling call
1024 19:48:29.092683 (+4013462us) ts_tablet_manager.cc:1151] Deleting tablet 7e747880c4e1425ba749201e624a083e
1024 19:48:29.092701 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.092858 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4013284}
W20251024 19:48:29.093365 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c327a6167cf43449deb0db222d887ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.093597 3661 raft_consensus.cc:493] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.093807 3661 raft_consensus.cc:515] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.094492 3661 leader_election.cc:290] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.095285 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0797d14e9fd846788164747523df0e2c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:29.096015 1586 leader_election.cc:343] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.096436 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0797d14e9fd846788164747523df0e2c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:29.096783 1508 raft_consensus.cc:2468] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:29.097294 1586 leader_election.cc:304] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.097651 3661 raft_consensus.cc:2804] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:29.097765 3661 raft_consensus.cc:493] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:29.097908 3661 raft_consensus.cc:3060] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.099570 1774 ts_tablet_manager.cc:1929] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:29.099750 1774 log.cc:1199] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/292c31e7d4ce411bb336a1405ca43ba5
I20251024 19:48:29.100117 1774 ts_tablet_manager.cc:1950] T 292c31e7d4ce411bb336a1405ca43ba5 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:29.101291 3661 raft_consensus.cc:515] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:29.101890 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 24ff2839ff1e48d5a2f84890ec25c75b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ff2839ff1e48d5a2f84890ec25c75b already in progress: creating tablet
I20251024 19:48:29.102171 3661 leader_election.cc:290] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.102967 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0797d14e9fd846788164747523df0e2c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:29.103642 1586 leader_election.cc:343] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.103994 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0797d14e9fd846788164747523df0e2c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:29.104230 1508 raft_consensus.cc:3060] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.105067 3686 raft_consensus.cc:493] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.105198 3686 raft_consensus.cc:515] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.105919 3686 leader_election.cc:290] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:29.106576 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebcc09d8e029451d8fb6b6601ad16087" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:29.107138 1447 leader_election.cc:343] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.108590 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebcc09d8e029451d8fb6b6601ad16087" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:29.108852 1648 raft_consensus.cc:2468] T ebcc09d8e029451d8fb6b6601ad16087 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:29.109354 1508 raft_consensus.cc:2468] T 0797d14e9fd846788164747523df0e2c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:29.109381 1444 leader_election.cc:304] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.109834 3686 raft_consensus.cc:2804] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:29.109831 1586 leader_election.cc:304] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.109910 3686 raft_consensus.cc:493] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:29.109977 3686 raft_consensus.cc:3060] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.110119 3661 raft_consensus.cc:2804] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:29.110275 3661 raft_consensus.cc:697] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:29.110553 3661 consensus_queue.cc:237] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.113950 3686 raft_consensus.cc:515] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.114866 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebcc09d8e029451d8fb6b6601ad16087" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:29.115064 1648 raft_consensus.cc:3060] T ebcc09d8e029451d8fb6b6601ad16087 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.116868 1767 tablet_bootstrap.cc:492] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:29.117107 1767 ts_tablet_manager.cc:1403] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.000s
I20251024 19:48:29.117913 1648 raft_consensus.cc:2468] T ebcc09d8e029451d8fb6b6601ad16087 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:29.117923 1767 raft_consensus.cc:359] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.118144 1767 raft_consensus.cc:385] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.118221 1767 raft_consensus.cc:740] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.118336 1444 leader_election.cc:304] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:29.118475 1767 consensus_queue.cc:260] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.118618 3679 raft_consensus.cc:2804] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:29.119370 3686 leader_election.cc:290] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:29.119503 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebcc09d8e029451d8fb6b6601ad16087" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:29.120383 1447 leader_election.cc:343] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.121484 1767 ts_tablet_manager.cc:1434] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251024 19:48:29.121881 1767 tablet_bootstrap.cc:492] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:29.129050 1767 tablet_bootstrap.cc:654] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:29.104671 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a43796bcdf7e433ca0241add9efb8483 has no consensus state
W20251024 19:48:29.138190 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 018d3e6aba6740f0bd82e8c83863bc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 018d3e6aba6740f0bd82e8c83863bc4e already in progress: creating tablet
W20251024 19:48:29.138651 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 7e747880c4e1425ba749201e624a083e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e747880c4e1425ba749201e624a083e already in progress: creating tablet
I20251024 19:48:29.139911 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1201) took 4039 ms. Trace:
I20251024 19:48:29.140056 1775 rpcz_store.cc:276] 1024 19:48:25.100780 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.100849 (+ 69us) service_pool.cc:225] Handling call
1024 19:48:29.139550 (+4038701us) ts_tablet_manager.cc:1151] Deleting tablet 8f46974e56c64a58a9c562b235ff8cfe
1024 19:48:29.139563 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.139880 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4038485}
I20251024 19:48:29.159776 1768 tablet_bootstrap.cc:492] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:29.149741 3757 raft_consensus.cc:493] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.161041 3757 raft_consensus.cc:515] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.161427 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1202) took 4060 ms. Trace:
I20251024 19:48:29.161540 1775 rpcz_store.cc:276] 1024 19:48:25.100911 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.102074 (+ 1163us) service_pool.cc:225] Handling call
1024 19:48:29.161221 (+4059147us) ts_tablet_manager.cc:1151] Deleting tablet 6c887b0df24c45d894ca10d2eda71444
1024 19:48:29.161236 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.161401 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4058966}
I20251024 19:48:29.162634 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b22b58404b8b441ba97ef536885a66e0" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:29.162844 1648 raft_consensus.cc:2468] T b22b58404b8b441ba97ef536885a66e0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:29.163256 1444 leader_election.cc:304] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:29.164520 3757 leader_election.cc:290] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:29.166122 3679 raft_consensus.cc:697] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:29.166618 3679 consensus_queue.cc:237] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.165268 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b22b58404b8b441ba97ef536885a66e0" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:29.168085 1768 ts_tablet_manager.cc:1403] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.167s user 0.004s sys 0.001s
W20251024 19:48:29.168495 1447 leader_election.cc:343] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.169034 1768 raft_consensus.cc:359] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.169260 1768 raft_consensus.cc:385] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.169330 1768 raft_consensus.cc:740] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.169646 1768 consensus_queue.cc:260] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.170445 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 292c31e7d4ce411bb336a1405ca43ba5 (table default.Table_2021df09852e4dd09dbf7b36a0a3ef5b [id=13f16de0ad1f43ac80639d3cf35fbb16]) successfully deleted
W20251024 19:48:29.170810 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 3e09c12bcfd14ab081ab309f9dec4afc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e09c12bcfd14ab081ab309f9dec4afc already in progress: creating tablet
W20251024 19:48:29.171142 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet a27237314b684588aba9934065d9337b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a27237314b684588aba9934065d9337b already in progress: creating tablet
W20251024 19:48:29.171473 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet bd2c43c6f497496d85cb35cc44f5ac8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd2c43c6f497496d85cb35cc44f5ac8e already in progress: creating tablet
W20251024 19:48:29.172104 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 8f46974e56c64a58a9c562b235ff8cfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f46974e56c64a58a9c562b235ff8cfe already in progress: creating tablet
I20251024 19:48:29.158722 1766 tablet_bootstrap.cc:492] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:29.172447 1766 ts_tablet_manager.cc:1403] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.109s user 0.002s sys 0.004s
I20251024 19:48:29.173359 1766 raft_consensus.cc:359] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:29.173638 1766 raft_consensus.cc:385] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.173763 1766 raft_consensus.cc:740] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.174103 1766 consensus_queue.cc:260] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:29.177148 1767 tablet_bootstrap.cc:492] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:29.177338 1767 ts_tablet_manager.cc:1403] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.000s
I20251024 19:48:29.179522 3661 raft_consensus.cc:493] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.179663 3661 raft_consensus.cc:515] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.180255 3661 leader_election.cc:290] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.184222 1767 raft_consensus.cc:359] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.184417 1767 raft_consensus.cc:385] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.184518 1767 raft_consensus.cc:740] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.185566 1776 ts_tablet_manager.cc:1929] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251024 19:48:29.186007 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 6c887b0df24c45d894ca10d2eda71444 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c887b0df24c45d894ca10d2eda71444 already in progress: creating tablet
I20251024 19:48:29.186767 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_de9a4e8e076649ce9e7aa790a4f1ba7f" } new_table_name: "default.Table_1ffe82b269b547ca938ef7eb9f4d8cc6" modify_external_catalogs: true
I20251024 19:48:29.189038 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_686d12c4c37f4292aa2c15c0ee048f1e" } modify_external_catalogs: true
I20251024 19:48:29.189329 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_686d12c4c37f4292aa2c15c0ee048f1e" } modify_external_catalogs: true
I20251024 19:48:29.170133 1774 tablet_replica.cc:333] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:29.181459 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc31e6c25ad14947af20abd766f12e78" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:29.190181 1774 raft_consensus.cc:2243] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:29.190384 1774 raft_consensus.cc:2272] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:29.191017 1774 ts_tablet_manager.cc:1916] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:29.189313 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_d65da5a2f7214fdca166986046157c60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.194226 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d65da5a2f7214fdca166986046157c60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.196182 3759 raft_consensus.cc:493] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.196414 3759 raft_consensus.cc:515] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:29.174538 1775 tablet_replica.cc:333] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:29.177692 3760 raft_consensus.cc:2804] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:29.197201 3760 raft_consensus.cc:493] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:29.197299 3760 raft_consensus.cc:3060] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.180881 1506 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc31e6c25ad14947af20abd766f12e78" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:29.197705 1506 raft_consensus.cc:2468] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:29.190877 3686 raft_consensus.cc:493] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.198042 3686 raft_consensus.cc:515] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.198331 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94b35252e63f4a8ba305cc029ddb5ae8" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:29.198580 3686 leader_election.cc:290] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:29.199136 1447 leader_election.cc:343] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.199352 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8e8dbca79f4d72a2de51f105b209e4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:29.199884 1447 leader_election.cc:343] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.200759 3759 leader_election.cc:290] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
W20251024 19:48:29.201411 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a403201fab014ea98734c3e80649e088 consensus state has no leader
I20251024 19:48:29.201995 3760 raft_consensus.cc:515] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.202565 3760 leader_election.cc:290] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:29.203423 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b22b58404b8b441ba97ef536885a66e0" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:29.203881 1447 leader_election.cc:343] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.205757 1775 raft_consensus.cc:2243] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:29.205984 1775 raft_consensus.cc:2272] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:29.206602 1775 ts_tablet_manager.cc:1916] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:29.218070 1768 ts_tablet_manager.cc:1434] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20251024 19:48:29.235410 1768 tablet_bootstrap.cc:492] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:29.237365 1768 tablet_bootstrap.cc:654] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.238865 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_23a248d91f5d45fbbf4bf104efd8bfb2" } new_table_name: "default.Table_cb09595c6cfd4573a50124f133855fdb" modify_external_catalogs: true
I20251024 19:48:29.238971 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_23ab85c60df54739b2f4245d7463f4b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.239521 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23ab85c60df54739b2f4245d7463f4b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:29.249243 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 27729f3779454bf9bf7de8f7728ca1a6 consensus state has no leader
I20251024 19:48:29.249696 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_6993ba61772248f4b2788ef4947af471"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.250267 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6993ba61772248f4b2788ef4947af471 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.254191 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_06c1aef3fa5e4a9faaa7dec1f3b9efe0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.254760 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06c1aef3fa5e4a9faaa7dec1f3b9efe0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.274772 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_db05043847df4217a12a2524e5dc05ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.275823 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db05043847df4217a12a2524e5dc05ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.280856 3694 raft_consensus.cc:493] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.281010 3694 raft_consensus.cc:515] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.281697 3694 leader_election.cc:290] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.283846 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_b1945a1bef6a40fb8dd92f10bcb2ff10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.284459 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1945a1bef6a40fb8dd92f10bcb2ff10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.186933 1767 consensus_queue.cc:260] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.288547 1767 ts_tablet_manager.cc:1434] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.111s user 0.003s sys 0.000s
I20251024 19:48:29.288888 1767 tablet_bootstrap.cc:492] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:29.290665 1767 tablet_bootstrap.cc:654] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:29.190640 1586 leader_election.cc:343] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:29.291558 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 100967us
I20251024 19:48:29.292161 1586 leader_election.cc:304] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.292491 3661 raft_consensus.cc:2804] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:29.199266 1648 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8e8dbca79f4d72a2de51f105b209e4" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:29.292630 3661 raft_consensus.cc:493] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:29.292733 3661 raft_consensus.cc:3060] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.200618 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94b35252e63f4a8ba305cc029ddb5ae8" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:29.203207 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b22b58404b8b441ba97ef536885a66e0" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:29.293654 1647 raft_consensus.cc:3060] T b22b58404b8b441ba97ef536885a66e0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.293632 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_69274ddf02254478b87008336a851c38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:29.293924 3404 raft_consensus.cc:493] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.294075 3404 raft_consensus.cc:515] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:29.294188 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69274ddf02254478b87008336a851c38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.294598 3404 leader_election.cc:290] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.295868 3661 raft_consensus.cc:515] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.296362 1647 raft_consensus.cc:2468] T b22b58404b8b441ba97ef536885a66e0 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:29.296751 3661 leader_election.cc:290] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.282667 1645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a69c0918d2e4eed9d91b1913ec2c21e" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:29.301129 3738 raft_consensus.cc:493] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251024 19:48:29.301263 1586 consensus_peers.cc:597] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83b91a5d764449d5b82ef6d1ee0360f5. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:29.301343 3738 raft_consensus.cc:515] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:29.301964 3738 leader_election.cc:290] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.302795 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b2992a738b440d798f5e61cc45ec088" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:29.303179 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc31e6c25ad14947af20abd766f12e78" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:29.303959 1586 leader_election.cc:343] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.304527 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6f8764e9ae472b9ac35a032182aed1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:29.304973 1586 leader_election.cc:343] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.309742 1776 log.cc:1199] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/ee3c521876324808857f7724028c895f
I20251024 19:48:29.310228 1776 ts_tablet_manager.cc:1950] T ee3c521876324808857f7724028c895f P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:29.311153 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1289) took 1956 ms. Trace:
I20251024 19:48:29.311265 1776 rpcz_store.cc:276] 1024 19:48:27.354200 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.354568 (+ 368us) service_pool.cc:225] Handling call
1024 19:48:29.046978 (+1692410us) ts_tablet_manager.cc:1151] Deleting tablet ee3c521876324808857f7724028c895f
1024 19:48:29.046990 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.182068 (+135078us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:29.185463 (+ 3395us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:29.311092 (+125629us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1692246}
I20251024 19:48:29.311702 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet ee3c521876324808857f7724028c895f (table default.Table_a86db858d09749a9a4706c761c20ee9a [id=f49bfd0c428d44f69a3d97a6c51a380a]) successfully deleted
I20251024 19:48:29.316282 1768 tablet_bootstrap.cc:492] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:29.318984 1768 ts_tablet_manager.cc:1403] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.084s user 0.004s sys 0.000s
I20251024 19:48:29.319754 1768 raft_consensus.cc:359] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.319928 1768 raft_consensus.cc:385] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.319994 1768 raft_consensus.cc:740] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.322036 1774 ts_tablet_manager.cc:1929] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:29.322144 1774 log.cc:1199] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/2d01572e51694d9bb58c3a59d4205102
I20251024 19:48:29.322458 1774 ts_tablet_manager.cc:1950] T 2d01572e51694d9bb58c3a59d4205102 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:29.323568 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1290) took 1968 ms. Trace:
I20251024 19:48:29.323961 1768 consensus_queue.cc:260] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.324528 1776 rpcz_store.cc:276] 1024 19:48:27.354314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.354766 (+ 452us) service_pool.cc:225] Handling call
1024 19:48:29.323070 (+1968304us) ts_tablet_manager.cc:1151] Deleting tablet 38818c7997c84cedad5f9b09eef77105
1024 19:48:29.323085 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.323180 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1968161}
W20251024 19:48:29.325058 1176 catalog_manager.cc:4952] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 38818c7997c84cedad5f9b09eef77105 because the tablet was not found. No further retry: Not found: Tablet not found: 38818c7997c84cedad5f9b09eef77105
I20251024 19:48:29.325305 1776 tablet_replica.cc:333] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:29.325538 1776 raft_consensus.cc:2243] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:29.325940 1776 raft_consensus.cc:2272] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:29.327453 1768 ts_tablet_manager.cc:1434] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251024 19:48:29.329083 1768 tablet_bootstrap.cc:492] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:29.332152 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 2d01572e51694d9bb58c3a59d4205102 (table default.Table_bf15d3c700774490b1860d3d8865a95e [id=2ea13e78137d47429d7cbe374d2113ef]) successfully deleted
I20251024 19:48:29.217458 1775 ts_tablet_manager.cc:1929] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:29.230193 1766 ts_tablet_manager.cc:1434] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.058s user 0.001s sys 0.001s
I20251024 19:48:29.332587 1775 log.cc:1199] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/64499c8734d34c66b0b52a2cd3ef0673
I20251024 19:48:29.332892 1766 tablet_bootstrap.cc:492] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:29.332949 1775 ts_tablet_manager.cc:1950] T 64499c8734d34c66b0b52a2cd3ef0673 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:29.333758 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1207) took 3965 ms. Trace:
I20251024 19:48:29.333853 1775 rpcz_store.cc:276] 1024 19:48:25.368709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.368765 (+ 56us) service_pool.cc:225] Handling call
1024 19:48:29.174472 (+3805707us) ts_tablet_manager.cc:1151] Deleting tablet 64499c8734d34c66b0b52a2cd3ef0673
1024 19:48:29.174487 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.214004 (+ 39517us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:29.217331 (+ 3327us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:29.333734 (+116403us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3805521}
I20251024 19:48:29.334296 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 64499c8734d34c66b0b52a2cd3ef0673 (table default.Table_f11b271966af455798227788699910b6 [id=b76ea2d1917f4b3cb23721c0df6d641f]) successfully deleted
I20251024 19:48:29.334645 1766 tablet_bootstrap.cc:654] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.283094 1506 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a69c0918d2e4eed9d91b1913ec2c21e" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:29.335558 1506 raft_consensus.cc:2393] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
W20251024 19:48:29.299376 1444 leader_election.cc:343] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.336755 1444 leader_election.cc:304] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
W20251024 19:48:29.337157 1444 leader_election.cc:343] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.337255 1444 leader_election.cc:304] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:29.300055 3686 consensus_queue.cc:1048] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:29.326565 1776 ts_tablet_manager.cc:1916] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:29.337899 3686 raft_consensus.cc:2749] T 3b8e8dbca79f4d72a2de51f105b209e4 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:29.338038 3686 raft_consensus.cc:2749] T 94b35252e63f4a8ba305cc029ddb5ae8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:29.301920 1503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc31e6c25ad14947af20abd766f12e78" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:29.338199 1503 raft_consensus.cc:3060] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.340875 1503 raft_consensus.cc:2468] T bc31e6c25ad14947af20abd766f12e78 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:29.341023 1768 tablet_bootstrap.cc:654] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.302114 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b2992a738b440d798f5e61cc45ec088" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:29.341390 1508 raft_consensus.cc:2393] T 2b2992a738b440d798f5e61cc45ec088 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af3952de050140d383f6b0de7666fa97 in current term 1: Already voted for candidate 43bec2f03fc849099e254f6335e9d532 in this term.
I20251024 19:48:29.304425 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6f8764e9ae472b9ac35a032182aed1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:29.341956 1507 raft_consensus.cc:2468] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:29.342737 1586 leader_election.cc:304] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.343106 1444 leader_election.cc:304] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.343183 1586 leader_election.cc:304] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.343470 3738 raft_consensus.cc:2804] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:29.343627 3738 raft_consensus.cc:697] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:29.343634 3760 raft_consensus.cc:2804] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:29.343600 1586 leader_election.cc:304] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.343746 3760 raft_consensus.cc:697] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:29.343995 3404 raft_consensus.cc:3060] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.343968 3760 consensus_queue.cc:237] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.344223 3661 raft_consensus.cc:2804] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:29.344347 3661 raft_consensus.cc:493] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:29.344501 3661 raft_consensus.cc:3060] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.346712 3404 raft_consensus.cc:2749] T 2b2992a738b440d798f5e61cc45ec088 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:29.347544 3661 raft_consensus.cc:515] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:29.347842 1308 leader_election.cc:304] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:29.347458 3738 consensus_queue.cc:237] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.349676 3694 raft_consensus.cc:3060] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.350741 3661 leader_election.cc:290] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.352959 3694 raft_consensus.cc:2749] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:29.353217 3661 raft_consensus.cc:493] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.353390 3661 raft_consensus.cc:515] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.353966 3661 leader_election.cc:290] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.358665 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6f8764e9ae472b9ac35a032182aed1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:29.358819 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d1b6eaa353c4284a2b114b42437272c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:29.358950 1507 raft_consensus.cc:3060] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.358999 1508 raft_consensus.cc:2468] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:29.359642 1586 leader_election.cc:304] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:29.365967 1507 raft_consensus.cc:2468] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:29.366518 3661 raft_consensus.cc:2804] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:29.366703 3661 raft_consensus.cc:493] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:29.366834 1586 leader_election.cc:304] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:29.366887 3661 raft_consensus.cc:3060] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.367009 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_54c5eb4772114f2e829fe31c91a9292d" } modify_external_catalogs: true
I20251024 19:48:29.367069 3573 raft_consensus.cc:2804] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:29.367220 3573 raft_consensus.cc:697] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:29.367190 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_54c5eb4772114f2e829fe31c91a9292d" } modify_external_catalogs: true
I20251024 19:48:29.367492 3573 consensus_queue.cc:237] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
W20251024 19:48:29.368477 1177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.252.253:41375: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20251024 19:48:29.370325 3661 raft_consensus.cc:515] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.371563 3661 leader_election.cc:290] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.372013 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d1b6eaa353c4284a2b114b42437272c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:29.372251 1507 raft_consensus.cc:3060] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.378460 1766 tablet_bootstrap.cc:492] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:29.378700 1766 ts_tablet_manager.cc:1403] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.010s
I20251024 19:48:29.386210 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_43953a13cfc64df2920d9de139b89be0" } new_table_name: "default.Table_148cbfa4cca1407a9d1b96f9f7249799" modify_external_catalogs: true
I20251024 19:48:29.383383 1767 tablet_bootstrap.cc:492] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:29.387058 1767 ts_tablet_manager.cc:1403] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.098s user 0.006s sys 0.004s
W20251024 19:48:29.388624 1177 consensus_peers.cc:597] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c -> Peer 82e82847f8904b59b03f446b7fce6233 (127.0.252.253:41375): Couldn't send request to peer 82e82847f8904b59b03f446b7fce6233. Status: Network error: Client connection negotiation failed: client connection to 127.0.252.253:41375: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251024 19:48:29.389129 1767 raft_consensus.cc:359] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:29.391441 1767 raft_consensus.cc:385] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.391527 1767 raft_consensus.cc:740] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.392153 1767 consensus_queue.cc:260] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:29.392717 1767 ts_tablet_manager.cc:1434] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251024 19:48:29.393035 1767 tablet_bootstrap.cc:492] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
W20251024 19:48:29.394570 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85c7134bb2f046319868d41df161572c consensus state has no leader
I20251024 19:48:29.395025 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9396a388f16442a59278d2bb67298c24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.395565 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9396a388f16442a59278d2bb67298c24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.402179 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_7040850ac4da4483ad7cc6d41cc17eb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.402952 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7040850ac4da4483ad7cc6d41cc17eb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.408005 1507 raft_consensus.cc:2468] T 8d1b6eaa353c4284a2b114b42437272c P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:29.410395 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_535b9a93bdda424a81bbdc6c08c3450f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.410916 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_535b9a93bdda424a81bbdc6c08c3450f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.412104 1586 leader_election.cc:304] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:29.412410 3661 raft_consensus.cc:2804] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:29.412590 3661 raft_consensus.cc:697] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:29.379701 1766 raft_consensus.cc:359] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.414029 1766 raft_consensus.cc:385] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.414103 1766 raft_consensus.cc:740] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.414369 1766 consensus_queue.cc:260] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.414623 3404 raft_consensus.cc:493] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.414739 3404 raft_consensus.cc:515] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.415346 3404 leader_election.cc:290] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:29.418275 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_4d0d05989b3844cd8813c527af4862a1" } new_table_name: "default.Table_fdb6e0fc5d034cd4a691a07b33911e90" modify_external_catalogs: true
I20251024 19:48:29.421887 1775 tablet_replica.cc:333] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:29.424015 3686 consensus_queue.cc:1048] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:29.426160 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1a7468b6f68046fa91ce076ffe3f97a5 consensus state has no leader
I20251024 19:48:29.426714 3694 consensus_queue.cc:1048] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:29.427840 1767 tablet_bootstrap.cc:654] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.444831 1775 raft_consensus.cc:2243] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:29.445333 1775 raft_consensus.cc:2272] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:29.447253 1775 ts_tablet_manager.cc:1916] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:29.447742 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_3ae09b04a7ef498b9ebe1867f056d9c9" } new_table_name: "default.Table_64ec0127348a487dbe59a85f297ba180" modify_external_catalogs: true
I20251024 19:48:29.446518 1766 ts_tablet_manager.cc:1434] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.068s user 0.001s sys 0.001s
I20251024 19:48:29.450915 1766 tablet_bootstrap.cc:492] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
W20251024 19:48:29.451509 1586 leader_election.cc:343] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Not found: Tablet not found: aa08555dbe1e4ad8968c534510eca9ff
I20251024 19:48:29.456024 3661 consensus_queue.cc:237] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:29.456197 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44305f9f044b4b14bbf66bbe612d14a1 consensus state has no leader
I20251024 19:48:29.456699 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_862cb2dda48a4dbd954b5d852fb0e5fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.457387 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_862cb2dda48a4dbd954b5d852fb0e5fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.460228 3782 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.252.253:41375 every 8 connection(s)
I20251024 19:48:29.460649 3466 rpc_server.cc:307] RPC server started. Bound to: 127.0.252.253:41375
I20251024 19:48:29.463037 3466 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/master-1/data/info.pb
I20251024 19:48:29.464033 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e845d12029a948f7b361c33a3cdabfc2" } new_table_name: "default.Table_269d8fea5d5744a79258124b3a7b61f4" modify_external_catalogs: true
I20251024 19:48:29.469463 1011 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskc9NInd/build/asan/bin/kudu as pid 3466
I20251024 19:48:29.472265 1767 tablet_bootstrap.cc:492] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:29.472399 1767 ts_tablet_manager.cc:1403] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.002s
I20251024 19:48:29.473055 1767 raft_consensus.cc:359] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.473225 1767 raft_consensus.cc:385] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.473268 1767 raft_consensus.cc:740] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.474393 1766 tablet_bootstrap.cc:654] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:29.474541 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9ee0844646e4034b45f307738623968 consensus state has no leader
I20251024 19:48:29.476496 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_0b75e1eb5e35490ab2947463e6788059"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:29.477030 1189 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet c1fc0f99ca5c42bfb9663e21a7b761c9 (table default.Table_d7da4325d747469b86ff66b85690867d [id=a32bd550682746f0a2861ff8be404992]). Expected version 1 got 0
I20251024 19:48:29.477267 1189 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 58a8f4f90f794e83a158e0026e5cea7f (table default.Table_c7b7dc69f2e640f6bb21208e8d8f32b8 [id=3fe81a9c4a134e908f4f2599c4b0cbf4]). Expected version 1 got 0
I20251024 19:48:29.478817 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2ca6f1a424034ecdb01cef12ae91756c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:29.478749 1187 catalog_manager.cc:5649] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43bec2f03fc849099e254f6335e9d532 (127.0.252.193). New cstate: current_term: 1 leader_uuid: "43bec2f03fc849099e254f6335e9d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:29.479106 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 2b2992a738b440d798f5e61cc45ec088 (table default.Table_8523c29419f0445aa4da2ffdc6480f1a [id=40394401a8a84431bfb5e2b1db774832]). Expected version 4 got 0
W20251024 19:48:29.479281 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ca6f1a424034ecdb01cef12ae91756c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.479341 1187 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet c1fc0f99ca5c42bfb9663e21a7b761c9 (table default.Table_d7da4325d747469b86ff66b85690867d [id=a32bd550682746f0a2861ff8be404992]). Expected version 1 got 0
W20251024 19:48:29.479995 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 043fac06e6324032a61c42a10d51ea22 consensus state has no leader
I20251024 19:48:29.484520 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 92d78642772d44179e67d75c51b9da4e (table default.Table_52aa7a7ba04d4184bc9c976c07387929 [id=37332f920f4d4827b16f1c1c7bffda6d]). Expected version 2 got 0
W20251024 19:48:29.477083 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b75e1eb5e35490ab2947463e6788059 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.490950 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 1b11c4f30236496e83f6336365cb5972
I20251024 19:48:29.490970 1767 consensus_queue.cc:260] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.492350 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b11c4f30236496e83f6336365cb5972 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.492918 1767 ts_tablet_manager.cc:1434] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20251024 19:48:29.493180 1767 tablet_bootstrap.cc:492] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:29.494072 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b11c4f30236496e83f6336365cb5972 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.481701 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 1295c99836d14a1cb0002e6c6aaee1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:07 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:29.492154 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 1b11c4f30236496e83f6336365cb5972 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:46462
W20251024 19:48:29.495194 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 1b11c4f30236496e83f6336365cb5972 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b11c4f30236496e83f6336365cb5972 already in progress: creating tablet
I20251024 19:48:29.502650 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 30bce965315d43948ecb8282c829c557 with delete_type TABLET_DATA_DELETED (replaced by tablet 043fac06e6324032a61c42a10d51ea22) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.503032 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 292c31e7d4ce411bb336a1405ca43ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:14 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.503255 1774 tablet_replica.cc:333] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:29.503448 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2f068c3ae13243888ceb972d31957bcb
I20251024 19:48:29.503461 1774 raft_consensus.cc:2243] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:29.503607 1774 raft_consensus.cc:2272] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:29.504159 1774 ts_tablet_manager.cc:1916] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:29.504382 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f068c3ae13243888ceb972d31957bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.504920 1767 tablet_bootstrap.cc:654] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.505374 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f068c3ae13243888ceb972d31957bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:29.508023 1183 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 58a8f4f90f794e83a158e0026e5cea7f (table default.Table_c7b7dc69f2e640f6bb21208e8d8f32b8 [id=3fe81a9c4a134e908f4f2599c4b0cbf4]). Expected version 1 got 0
I20251024 19:48:29.508222 1183 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 25ced70ff40d4149bb92e8b3767ec38f (table default.Table_693aa7a74dc241f9af879ec2c4cfd808 [id=eacc2d37d57f4d588101cb545db90c7d]). Expected version 1 got 0
I20251024 19:48:29.510638 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 94b35252e63f4a8ba305cc029ddb5ae8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b97907234e543c2806f40619046f6ae) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:29.510812 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 2671108031db449cae2135835fc5f739 (table default.Table_8514e876be154443ad1f2827119421c5 [id=2f6b7b685af84337bd64141ef2a9e992]). Expected version 3 got 0
I20251024 19:48:29.511097 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 6ef2add387da44868dbd2b5724ee7d86 (table default.Table_256d5a49b85c4a07b9c5e1839ee9bee6 [id=e55575a8212b40e093444ad77f9835e5]). Expected version 1 got 0
I20251024 19:48:29.511302 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 2b2992a738b440d798f5e61cc45ec088 (table default.Table_8523c29419f0445aa4da2ffdc6480f1a [id=40394401a8a84431bfb5e2b1db774832]). Expected version 4 got 0
I20251024 19:48:29.511515 1191 catalog_manager.cc:5649] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:29.511855 1191 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 0797d14e9fd846788164747523df0e2c (table default.Table_1f866642ce4a4ee885f169c4815bd1fb [id=a07755e138c84fd18c0fe72e81ab758b]). Expected version 1 got 0
I20251024 19:48:29.505743 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f068c3ae13243888ceb972d31957bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.515012 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 489752031a5a4d168367cb4d143702b7
I20251024 19:48:29.516139 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 489752031a5a4d168367cb4d143702b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.516338 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 489752031a5a4d168367cb4d143702b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:29.516903 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 489752031a5a4d168367cb4d143702b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:29.518616 1308 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 170803us
I20251024 19:48:29.520897 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6f8764e9ae472b9ac35a032182aed1" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:29.521718 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d1b6eaa353c4284a2b114b42437272c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:29.522284 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_ef5094e87a7a4ee1b8f82f1d2943e7d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:29.522353 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d1b6eaa353c4284a2b114b42437272c" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:29.522807 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef5094e87a7a4ee1b8f82f1d2943e7d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.523669 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa08555dbe1e4ad8968c534510eca9ff" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:29.524255 1586 leader_election.cc:343] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:29.524857 1586 leader_election.cc:343] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:29.525480 1586 leader_election.cc:343] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.526769 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 830528cabe344ce89e5092dae613c7fe
W20251024 19:48:29.526779 1586 leader_election.cc:343] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.526870 1586 leader_election.cc:304] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3952de050140d383f6b0de7666fa97; no voters: 10a9536207e343f188e0ef709081d04e, 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.527189 3661 raft_consensus.cc:2749] T aa08555dbe1e4ad8968c534510eca9ff P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:29.527654 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 830528cabe344ce89e5092dae613c7fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.528347 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 830528cabe344ce89e5092dae613c7fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.529678 3694 consensus_queue.cc:1048] T 2b2992a738b440d798f5e61cc45ec088 P 43bec2f03fc849099e254f6335e9d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:29.539588 1767 tablet_bootstrap.cc:492] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:29.539812 1767 ts_tablet_manager.cc:1403] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.003s
I20251024 19:48:29.540591 1767 raft_consensus.cc:359] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.540835 1767 raft_consensus.cc:385] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.540940 1767 raft_consensus.cc:740] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.541229 1767 consensus_queue.cc:260] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.541744 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 830528cabe344ce89e5092dae613c7fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:29.541865 1767 ts_tablet_manager.cc:1434] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 19:48:29.542577 1767 tablet_bootstrap.cc:492] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:29.544436 3759 raft_consensus.cc:493] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af3952de050140d383f6b0de7666fa97)
I20251024 19:48:29.544571 3759 raft_consensus.cc:515] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.544796 1767 tablet_bootstrap.cc:654] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.547924 3759 leader_election.cc:290] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:29.547974 3686 raft_consensus.cc:493] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.548108 3686 raft_consensus.cc:515] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.548612 3686 leader_election.cc:290] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
W20251024 19:48:29.549221 1444 leader_election.cc:343] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Not found: Tablet not found: 4100abc9ff12471e9a4959b6b24f7e99
I20251024 19:48:29.549867 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ec11453f49054f9dbbe620374e59a6ce
I20251024 19:48:29.549844 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24cb7ec9294d4f438a042451f669b98f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:29.550644 1447 leader_election.cc:343] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Not found: Tablet not found: 4100abc9ff12471e9a4959b6b24f7e99
I20251024 19:48:29.550657 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24cb7ec9294d4f438a042451f669b98f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:29.550765 1447 leader_election.cc:304] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:29.550920 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet ec11453f49054f9dbbe620374e59a6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.551121 3760 raft_consensus.cc:2749] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251024 19:48:29.551421 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet ec11453f49054f9dbbe620374e59a6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:54174
W20251024 19:48:29.551892 1444 leader_election.cc:343] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:29.552645 1447 leader_election.cc:343] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.552767 1447 leader_election.cc:304] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:29.552811 3799 tablet.cc:1722] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:29.553206 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet ec11453f49054f9dbbe620374e59a6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:28 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:29.553558 3686 raft_consensus.cc:2749] T 24cb7ec9294d4f438a042451f669b98f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:29.556442 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 0a1f3aef6a1d4207b2987c44f1452f79 (DEFAULT_TABLE table=default.Table_14a8b750c14d45f0af303e5b5202859e [id=e1019019399e491ba2bb38e514bef638]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.558557 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 0a1f3aef6a1d4207b2987c44f1452f79 (DEFAULT_TABLE table=default.Table_14a8b750c14d45f0af303e5b5202859e [id=e1019019399e491ba2bb38e514bef638]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.559051 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a1f3aef6a1d4207b2987c44f1452f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.562052 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d1b6eaa353c4284a2b114b42437272c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:19 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.562569 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet aa08555dbe1e4ad8968c534510eca9ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:21 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.571395 1767 tablet_bootstrap.cc:492] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:29.571573 1767 ts_tablet_manager.cc:1403] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.002s
I20251024 19:48:29.572144 1623 tablet_service.cc:1552] Processing DeleteTablet for tablet 15de0d8aa9e545c09233186bbe361b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.572155 1617 tablet_service.cc:1552] Processing DeleteTablet for tablet 7deb97be82964ea0b2fe9caa867a6460 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.572273 1767 raft_consensus.cc:359] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.572453 1767 raft_consensus.cc:385] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.572508 1767 raft_consensus.cc:740] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.572706 1767 consensus_queue.cc:260] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.572768 3573 raft_consensus.cc:493] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.572952 3573 raft_consensus.cc:515] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.573221 1767 ts_tablet_manager.cc:1434] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:29.573473 1767 tablet_bootstrap.cc:492] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:29.573444 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 2b2992a738b440d798f5e61cc45ec088 (table default.Table_8523c29419f0445aa4da2ffdc6480f1a [id=40394401a8a84431bfb5e2b1db774832]). Expected version 4 got 0
I20251024 19:48:29.573566 3573 leader_election.cc:290] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:29.573779 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 58a8f4f90f794e83a158e0026e5cea7f (table default.Table_c7b7dc69f2e640f6bb21208e8d8f32b8 [id=3fe81a9c4a134e908f4f2599c4b0cbf4]). Expected version 1 got 0
I20251024 19:48:29.574009 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet c1fc0f99ca5c42bfb9663e21a7b761c9 (table default.Table_d7da4325d747469b86ff66b85690867d [id=a32bd550682746f0a2861ff8be404992]). Expected version 1 got 0
I20251024 19:48:29.574622 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet a1fbbba88a294adfbf97c0343d86698e (table default.Table_7bc8cd4a930b44dfafa473297875594d [id=a60c8222aa0248c091fab8dc34100847]). Expected version 2 got 0
I20251024 19:48:29.574662 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ef2add387da44868dbd2b5724ee7d86" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:29.574865 1508 raft_consensus.cc:2468] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:29.575424 1586 leader_election.cc:304] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:29.575798 3661 raft_consensus.cc:2804] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:29.575878 3661 raft_consensus.cc:493] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:29.575950 3661 raft_consensus.cc:3060] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.578752 3661 raft_consensus.cc:515] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.579301 3661 leader_election.cc:290] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:29.579535 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 64499c8734d34c66b0b52a2cd3ef0673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:09 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.580484 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ef2add387da44868dbd2b5724ee7d86" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:29.580662 1508 raft_consensus.cc:3060] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.580849 1767 tablet_bootstrap.cc:654] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.584180 1508 raft_consensus.cc:2468] T 6ef2add387da44868dbd2b5724ee7d86 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:29.584623 1586 leader_election.cc:304] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:29.584944 3661 raft_consensus.cc:2804] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:29.585074 3661 raft_consensus.cc:697] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:29.585285 3661 consensus_queue.cc:237] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.585878 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet ab488e4971874a16b2e052c84c250777 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.586143 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a69c0918d2e4eed9d91b1913ec2c21e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.589262 1192 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 774a2de2e3764c54b72acfb952ed3e11 (table default.Table_5fb8e365cba24ded8061c38d9c695ff8 [id=1a8ca3395d7b471fa354154f2468080f]). Expected version 1 got 0
I20251024 19:48:29.590338 1766 tablet_bootstrap.cc:492] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:29.594045 1766 ts_tablet_manager.cc:1403] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.143s user 0.005s sys 0.001s
I20251024 19:48:29.596393 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 2b2992a738b440d798f5e61cc45ec088 (table default.Table_8523c29419f0445aa4da2ffdc6480f1a [id=40394401a8a84431bfb5e2b1db774832]). Expected version 4 got 0
I20251024 19:48:29.598275 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c929febd979f4b6f85071cd5852e90e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.598980 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c929febd979f4b6f85071cd5852e90e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.601941 1768 tablet_bootstrap.cc:492] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:29.604785 1768 ts_tablet_manager.cc:1403] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.275s user 0.003s sys 0.002s
I20251024 19:48:29.606787 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ef2add387da44868dbd2b5724ee7d86" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:29.607402 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ef2add387da44868dbd2b5724ee7d86" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:29.607548 1183 master_service.cc:954] ReplaceTablet: received request to replace tablet 300952047a324a25a7c687eff20d6e56 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:29.607844 1190 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_69b9df2416fe4680b34e9b399ef39216" } new_table_name: "default.Table_5ca306bc7a244de7b3f5d42c7b831539" modify_external_catalogs: true
I20251024 19:48:29.608870 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a1f3aef6a1d4207b2987c44f1452f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.609112 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_cc2d8128e49f486bbc9cf83bb4b112cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.609613 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc2d8128e49f486bbc9cf83bb4b112cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.624197 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 300952047a324a25a7c687eff20d6e56
I20251024 19:48:29.624743 1183 catalog_manager.cc:6719] ReplaceTablet: tablet 300952047a324a25a7c687eff20d6e56 deleted and replaced by tablet be6a55e263ce4e9d85ddb2443e5a8426
I20251024 19:48:29.625381 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 300952047a324a25a7c687eff20d6e56 with delete_type TABLET_DATA_DELETED (replaced by tablet be6a55e263ce4e9d85ddb2443e5a8426) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:29.625964 1586 leader_election.cc:343] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:29.626068 1190 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b445098bfd04729808641fcfb76927d consensus state has no leader
W20251024 19:48:29.626446 1586 leader_election.cc:343] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:29.627154 1586 consensus_peers.cc:597] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83b91a5d764449d5b82ef6d1ee0360f5. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:29.627437 1586 consensus_peers.cc:597] T 2d01572e51694d9bb58c3a59d4205102 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d01572e51694d9bb58c3a59d4205102. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:29.629654 3760 raft_consensus.cc:493] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.629802 3760 raft_consensus.cc:515] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.630401 3760 leader_election.cc:290] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:29.631541 1647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92d78642772d44179e67d75c51b9da4e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:29.631692 1647 raft_consensus.cc:2468] T 92d78642772d44179e67d75c51b9da4e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 0.
I20251024 19:48:29.632153 1444 leader_election.cc:304] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:29.632622 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92d78642772d44179e67d75c51b9da4e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:29.632684 3804 tablet.cc:1722] T c1fc0f99ca5c42bfb9663e21a7b761c9 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251024 19:48:29.633056 1447 leader_election.cc:343] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.633364 3760 raft_consensus.cc:2804] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:29.633427 3760 raft_consensus.cc:493] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:29.633498 3760 raft_consensus.cc:3060] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.634738 1337 tablet_service.cc:1552] Processing DeleteTablet for tablet 300952047a324a25a7c687eff20d6e56 with delete_type TABLET_DATA_DELETED (replaced by tablet be6a55e263ce4e9d85ddb2443e5a8426) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.635088 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet 300952047a324a25a7c687eff20d6e56 with delete_type TABLET_DATA_DELETED (replaced by tablet be6a55e263ce4e9d85ddb2443e5a8426) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:29.636404 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_353d0c63ddcc41d2aa242bb7e8f89f14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.636929 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_353d0c63ddcc41d2aa242bb7e8f89f14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.636950 3760 raft_consensus.cc:515] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.638422 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92d78642772d44179e67d75c51b9da4e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:29.638937 1447 leader_election.cc:343] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.639132 3760 leader_election.cc:290] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Requested vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:29.639447 1645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92d78642772d44179e67d75c51b9da4e" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97"
I20251024 19:48:29.639674 1645 raft_consensus.cc:3060] T 92d78642772d44179e67d75c51b9da4e P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.642803 1645 raft_consensus.cc:2468] T 92d78642772d44179e67d75c51b9da4e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10a9536207e343f188e0ef709081d04e in term 1.
I20251024 19:48:29.645918 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_92a5692084bf4d7c9caf7f90960e8f9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.646503 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92a5692084bf4d7c9caf7f90960e8f9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.648705 1444 leader_election.cc:304] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.649362 3760 raft_consensus.cc:2804] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:29.649690 3760 raft_consensus.cc:697] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Becoming Leader. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Running, Role: LEADER
I20251024 19:48:29.649890 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_2fc89eee3d764c409f66027b6b878714"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:29.650002 3760 consensus_queue.cc:237] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:29.650441 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fc89eee3d764c409f66027b6b878714 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.653313 1766 raft_consensus.cc:359] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.653479 1766 raft_consensus.cc:385] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.653535 1766 raft_consensus.cc:740] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.653812 1766 consensus_queue.cc:260] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.654382 1766 ts_tablet_manager.cc:1434] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20251024 19:48:29.654482 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_8b66ef90eef047bfbf665db21ee48274" } new_table_name: "default.Table_1871aba06fbd431a8cebf906fbd2f854" modify_external_catalogs: true
I20251024 19:48:29.654656 1766 tablet_bootstrap.cc:492] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:29.655905 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_81498524d757401faaff766b9db57c27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:29.656383 1766 tablet_bootstrap.cc:654] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:29.656507 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81498524d757401faaff766b9db57c27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.662436 1774 ts_tablet_manager.cc:1929] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:29.662644 1774 log.cc:1199] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/30bce965315d43948ecb8282c829c557
I20251024 19:48:29.665004 1617 tablet_service.cc:1505] Processing CreateTablet for tablet f490ae985f544131a9ba8ecad55de647 (DEFAULT_TABLE table=default.Table_fb06db3f905c4de09f11a853224626b5 [id=091b90e5fc1f43659419c76f004105ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.665419 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f490ae985f544131a9ba8ecad55de647. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.669020 3790 sys_catalog.cc:263] Verifying existing consensus state
I20251024 19:48:29.669943 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 0a1f3aef6a1d4207b2987c44f1452f79 (DEFAULT_TABLE table=default.Table_14a8b750c14d45f0af303e5b5202859e [id=e1019019399e491ba2bb38e514bef638]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.670344 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a1f3aef6a1d4207b2987c44f1452f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.672087 1624 tablet_service.cc:1505] Processing CreateTablet for tablet c100afae6bfc4c38ad26bc6771b9214a (DEFAULT_TABLE table=default.Table_fa228a9956134b5d823ae8abd11821a0 [id=7c2e18e00f534f6d94838a4b5254b640]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.672421 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c100afae6bfc4c38ad26bc6771b9214a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.676062 1776 ts_tablet_manager.cc:1929] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:29.676193 1776 log.cc:1199] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/e38ef2ac19d14f9c91fa8abe86cc7987
I20251024 19:48:29.676558 1776 ts_tablet_manager.cc:1950] T e38ef2ac19d14f9c91fa8abe86cc7987 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:29.677091 3790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233: Bootstrap starting.
I20251024 19:48:29.677465 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1292) took 2322 ms. Trace:
I20251024 19:48:29.677820 3800 tablet.cc:1722] T c1fc0f99ca5c42bfb9663e21a7b761c9 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:29.686018 1775 ts_tablet_manager.cc:1929] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:29.686249 1775 log.cc:1199] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/786fe2fd6fca407e83a76f74f96a8b16
I20251024 19:48:29.686676 1775 ts_tablet_manager.cc:1950] T 786fe2fd6fca407e83a76f74f96a8b16 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:29.687577 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1208) took 4318 ms. Trace:
I20251024 19:48:29.687742 1775 rpcz_store.cc:276] 1024 19:48:25.369222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.369274 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:29.421830 (+4052556us) ts_tablet_manager.cc:1151] Deleting tablet 786fe2fd6fca407e83a76f74f96a8b16
1024 19:48:29.421842 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.500887 (+ 79045us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:29.685896 (+185009us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:29.687537 (+ 1641us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4048097}
I20251024 19:48:29.688391 1775 tablet_replica.cc:333] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:29.688681 1775 raft_consensus.cc:2243] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:29.688963 1775 raft_consensus.cc:2272] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:29.689632 1775 ts_tablet_manager.cc:1916] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:29.691743 1768 raft_consensus.cc:359] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:29.692051 1768 raft_consensus.cc:385] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.692162 1768 raft_consensus.cc:740] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.694165 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 786fe2fd6fca407e83a76f74f96a8b16 (table default.Table_03318e0d18d34db48da05226fd40029a [id=f86c9d168b0e452899202f63a090f87a]) successfully deleted
I20251024 19:48:29.694787 1774 ts_tablet_manager.cc:1950] T 30bce965315d43948ecb8282c829c557 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:29.695626 1337 tablet_service.cc:1505] Processing CreateTablet for tablet c100afae6bfc4c38ad26bc6771b9214a (DEFAULT_TABLE table=default.Table_fa228a9956134b5d823ae8abd11821a0 [id=7c2e18e00f534f6d94838a4b5254b640]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:29.700449 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 164831us
I20251024 19:48:29.706123 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c100afae6bfc4c38ad26bc6771b9214a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.706924 1334 tablet_service.cc:1505] Processing CreateTablet for tablet f490ae985f544131a9ba8ecad55de647 (DEFAULT_TABLE table=default.Table_fb06db3f905c4de09f11a853224626b5 [id=091b90e5fc1f43659419c76f004105ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.708442 1185 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a316e0ba22744f2c9b7431da3dbed677" } new_table_name: "default.Table_c57cf066b104499a8c0aa12258fa1ca0" modify_external_catalogs: true
I20251024 19:48:29.711844 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9ff460fc04ce46c0a4ae4163efee611b" } modify_external_catalogs: true
I20251024 19:48:29.711887 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_df274196c7264f7aa9c5e59b7b50d657"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:29.712004 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_9ff460fc04ce46c0a4ae4163efee611b" } modify_external_catalogs: true
W20251024 19:48:29.712409 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df274196c7264f7aa9c5e59b7b50d657 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.713671 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_25169e6ef41048cab53f651e950e992f" } modify_external_catalogs: true
I20251024 19:48:29.713812 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_25169e6ef41048cab53f651e950e992f" } modify_external_catalogs: true
I20251024 19:48:29.715237 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_a86884fdc4364764911f8b802fc35102" } new_table_name: "default.Table_e97cbcd597f2420d99b0c790803e4dff" modify_external_catalogs: true
I20251024 19:48:29.715283 3799 tablet.cc:1722] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:29.716524 1184 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_e26d99c098b743d2b81cfe7db40784af" } new_table_name: "default.Table_351d5526afcd4c308d56fe295092902f" modify_external_catalogs: true
I20251024 19:48:29.717805 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_c5daa131860c4f369dae0280b46a2165"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.718408 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5daa131860c4f369dae0280b46a2165 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.719488 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f490ae985f544131a9ba8ecad55de647. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.721335 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 2b2992a738b440d798f5e61cc45ec088 (table default.Table_8523c29419f0445aa4da2ffdc6480f1a [id=40394401a8a84431bfb5e2b1db774832]). Expected version 4 got 0
I20251024 19:48:29.721625 1191 catalog_manager.cc:5649] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 10a9536207e343f188e0ef709081d04e (127.0.252.194). New cstate: current_term: 1 leader_uuid: "10a9536207e343f188e0ef709081d04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: HEALTHY } } }
I20251024 19:48:29.721969 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet 92d78642772d44179e67d75c51b9da4e (table default.Table_52aa7a7ba04d4184bc9c976c07387929 [id=37332f920f4d4827b16f1c1c7bffda6d]). Expected version 2 got 0
I20251024 19:48:29.722165 1191 catalog_manager.cc:5741] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045) does not have the latest schema for tablet e0d5d210d8d8499cbfdbdaa29e5bb40b (table default.Table_1871aba06fbd431a8cebf906fbd2f854 [id=a2adb68f20424f9a8b8d8785496b2f3f]). Expected version 1 got 0
I20251024 19:48:29.723908 1774 tablet_replica.cc:333] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:29.724176 1774 raft_consensus.cc:2243] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:29.724328 1774 raft_consensus.cc:2272] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:29.726854 1774 ts_tablet_manager.cc:1916] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:29.746894 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 30bce965315d43948ecb8282c829c557 (table default.Table_82ab16da2ef848e098a10530d7d6f215 [id=d9ae61ad6f8545d8a9945f08bdd731a3]) successfully deleted
W20251024 19:48:29.747191 1179 catalog_manager.cc:4952] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 292c31e7d4ce411bb336a1405ca43ba5 because the tablet was not found. No further retry: Not found: Tablet not found: 292c31e7d4ce411bb336a1405ca43ba5
W20251024 19:48:29.747398 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2f068c3ae13243888ceb972d31957bcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f068c3ae13243888ceb972d31957bcb already in progress: creating tablet
W20251024 19:48:29.747722 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 489752031a5a4d168367cb4d143702b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 489752031a5a4d168367cb4d143702b7 already in progress: creating tablet
W20251024 19:48:29.747946 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 830528cabe344ce89e5092dae613c7fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 830528cabe344ce89e5092dae613c7fe already in progress: creating tablet
W20251024 19:48:29.748124 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet ec11453f49054f9dbbe620374e59a6ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec11453f49054f9dbbe620374e59a6ce already in progress: creating tablet
I20251024 19:48:29.751878 1774 ts_tablet_manager.cc:1929] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:29.752216 1774 log.cc:1199] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/64499c8734d34c66b0b52a2cd3ef0673
I20251024 19:48:29.752687 1774 ts_tablet_manager.cc:1950] T 64499c8734d34c66b0b52a2cd3ef0673 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:29.753594 1768 consensus_queue.cc:260] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:29.754297 1768 ts_tablet_manager.cc:1434] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.148s user 0.002s sys 0.000s
I20251024 19:48:29.754599 1768 tablet_bootstrap.cc:492] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:29.756071 1768 tablet_bootstrap.cc:654] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.763437 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 64499c8734d34c66b0b52a2cd3ef0673 (table default.Table_f11b271966af455798227788699910b6 [id=b76ea2d1917f4b3cb23721c0df6d641f]) successfully deleted
I20251024 19:48:29.765375 1188 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 774a2de2e3764c54b72acfb952ed3e11 (table default.Table_5fb8e365cba24ded8061c38d9c695ff8 [id=1a8ca3395d7b471fa354154f2468080f]). Expected version 1 got 0
I20251024 19:48:29.765744 1188 catalog_manager.cc:5649] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:29.766109 1188 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 6ef2add387da44868dbd2b5724ee7d86 (table default.Table_256d5a49b85c4a07b9c5e1839ee9bee6 [id=e55575a8212b40e093444ad77f9835e5]). Expected version 1 got 0
I20251024 19:48:29.768473 3661 raft_consensus.cc:493] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.768636 3661 raft_consensus.cc:515] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.769244 3661 leader_election.cc:290] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.769982 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7deb97be82964ea0b2fe9caa867a6460" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:29.770772 1586 leader_election.cc:343] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:29.774590 1586 consensus_peers.cc:597] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64499c8734d34c66b0b52a2cd3ef0673. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:29.775233 1617 tablet_service.cc:1505] Processing CreateTablet for tablet de95bc9605e2423abf108f4be73cc8b8 (DEFAULT_TABLE table=default.Table_aadd52f1943441bda187b984246fcc02 [id=62e3c1cde81741718077889e71dab17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.776224 1623 tablet_service.cc:1505] Processing CreateTablet for tablet cb269d7648f240f8b8b5827f0467e92f (DEFAULT_TABLE table=default.Table_db369541ede24bd69d04163bb5d48d62 [id=cc40491291f246aa93c24a4bb08f5f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.776645 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb269d7648f240f8b8b5827f0467e92f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.776224 1612 tablet_service.cc:1505] Processing CreateTablet for tablet be72a342153d4abbb3a29e19a9f0c82b (DEFAULT_TABLE table=default.Table_8927bbd785744aa199fa84cc4af4efce [id=929b0a75d0e047639874eed0ad29a3d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.777941 1342 tablet_service.cc:1505] Processing CreateTablet for tablet be72a342153d4abbb3a29e19a9f0c82b (DEFAULT_TABLE table=default.Table_8927bbd785744aa199fa84cc4af4efce [id=929b0a75d0e047639874eed0ad29a3d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.778366 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be72a342153d4abbb3a29e19a9f0c82b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.781311 1419 heartbeater.cc:344] Connected to a master server at 127.0.252.253:41375
I20251024 19:48:29.782676 1697 heartbeater.cc:344] Connected to a master server at 127.0.252.253:41375
I20251024 19:48:29.783880 3661 consensus_queue.cc:1048] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:29.786785 3661 raft_consensus.cc:493] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.786929 3661 raft_consensus.cc:515] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.787609 3661 leader_election.cc:290] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.695113 1475 tablet_service.cc:1505] Processing CreateTablet for tablet c100afae6bfc4c38ad26bc6771b9214a (DEFAULT_TABLE table=default.Table_fa228a9956134b5d823ae8abd11821a0 [id=7c2e18e00f534f6d94838a4b5254b640]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.714030 3804 tablet.cc:1722] T e0d5d210d8d8499cbfdbdaa29e5bb40b P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:29.769896 3760 raft_consensus.cc:493] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.788097 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c100afae6bfc4c38ad26bc6771b9214a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.788026 3760 raft_consensus.cc:515] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.788645 3760 leader_election.cc:290] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:29.771195 1503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7deb97be82964ea0b2fe9caa867a6460" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:29.789062 1503 raft_consensus.cc:2468] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:29.789492 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2671108031db449cae2135835fc5f739" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
W20251024 19:48:29.789731 1185 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcee4c253eb84b9ab13cb664c4668a54 consensus state has no leader
I20251024 19:48:29.789837 1507 raft_consensus.cc:2468] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:29.774085 1490 tablet_service.cc:1505] Processing CreateTablet for tablet f490ae985f544131a9ba8ecad55de647 (DEFAULT_TABLE table=default.Table_fb06db3f905c4de09f11a853224626b5 [id=091b90e5fc1f43659419c76f004105ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.790360 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4ec69ffe2c024d1a8596a93183d1458b
I20251024 19:48:29.790577 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f490ae985f544131a9ba8ecad55de647. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.775362 1485 tablet_service.cc:1505] Processing CreateTablet for tablet de95bc9605e2423abf108f4be73cc8b8 (DEFAULT_TABLE table=default.Table_aadd52f1943441bda187b984246fcc02 [id=62e3c1cde81741718077889e71dab17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.776742 1479 tablet_service.cc:1505] Processing CreateTablet for tablet cb269d7648f240f8b8b5827f0467e92f (DEFAULT_TABLE table=default.Table_db369541ede24bd69d04163bb5d48d62 [id=cc40491291f246aa93c24a4bb08f5f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.776750 1473 tablet_service.cc:1505] Processing CreateTablet for tablet be72a342153d4abbb3a29e19a9f0c82b (DEFAULT_TABLE table=default.Table_8927bbd785744aa199fa84cc4af4efce [id=929b0a75d0e047639874eed0ad29a3d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.791814 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de95bc9605e2423abf108f4be73cc8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.791985 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25ced70ff40d4149bb92e8b3767ec38f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:29.793401 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 182314f0f3074150b139cb2476976e1a
I20251024 19:48:29.782392 1558 heartbeater.cc:344] Connected to a master server at 127.0.252.253:41375
I20251024 19:48:29.795462 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb269d7648f240f8b8b5827f0467e92f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.797822 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be72a342153d4abbb3a29e19a9f0c82b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:29.797770 1444 leader_election.cc:343] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.798556 1586 leader_election.cc:304] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.799057 1586 leader_election.cc:304] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:29.799345 3573 raft_consensus.cc:2804] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:29.799409 3573 raft_consensus.cc:493] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:29.799489 3573 raft_consensus.cc:3060] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.801832 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2671108031db449cae2135835fc5f739" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:29.803526 3738 raft_consensus.cc:2804] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:29.803618 3738 raft_consensus.cc:493] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251024 19:48:29.803591 1586 leader_election.cc:343] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.803689 3738 raft_consensus.cc:3060] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.677554 1776 rpcz_store.cc:276] 1024 19:48:27.354517 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.354941 (+ 424us) service_pool.cc:225] Handling call
1024 19:48:29.325217 (+1970276us) ts_tablet_manager.cc:1151] Deleting tablet e38ef2ac19d14f9c91fa8abe86cc7987
1024 19:48:29.325227 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.520756 (+195529us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:29.675963 (+155207us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:29.677443 (+ 1480us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1970148}
I20251024 19:48:29.806102 3573 raft_consensus.cc:515] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.807086 3738 raft_consensus.cc:515] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.807294 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ec69ffe2c024d1a8596a93183d1458b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.807365 3573 leader_election.cc:290] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.807592 3738 leader_election.cc:290] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.807811 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet e38ef2ac19d14f9c91fa8abe86cc7987 (table default.Table_6bb9ce85a6e341e7b2cc6d9ffc2c0127 [id=50a4f64cc7504ea190832dd265c81caf]) successfully deleted
I20251024 19:48:29.808506 1503 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2671108031db449cae2135835fc5f739" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:29.808674 1503 raft_consensus.cc:3060] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.808715 1767 tablet_bootstrap.cc:492] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:29.808835 1767 ts_tablet_manager.cc:1403] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.235s user 0.009s sys 0.012s
I20251024 19:48:29.809541 1767 raft_consensus.cc:359] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.809723 1767 raft_consensus.cc:385] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.809772 1767 raft_consensus.cc:740] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.809962 1767 consensus_queue.cc:260] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.810433 1767 ts_tablet_manager.cc:1434] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251024 19:48:29.811493 1184 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f10340e106b473eb715b74b129835b5 consensus state has no leader
I20251024 19:48:29.811700 1503 raft_consensus.cc:2468] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:29.811739 1190 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_d7da4325d747469b86ff66b85690867d [id=a32bd550682746f0a2861ff8be404992] alter complete (version 1)
I20251024 19:48:29.812032 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7deb97be82964ea0b2fe9caa867a6460" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:29.812188 1507 raft_consensus.cc:3060] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.813150 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25ced70ff40d4149bb92e8b3767ec38f" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:29.814250 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1293) took 2084 ms. Trace:
I20251024 19:48:29.814394 1776 rpcz_store.cc:276] 1024 19:48:27.729841 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.729904 (+ 63us) service_pool.cc:225] Handling call
1024 19:48:29.806447 (+2076543us) ts_tablet_manager.cc:1151] Deleting tablet b95e41141f5f4a5bbc39fb977d82252b
1024 19:48:29.814069 (+ 7622us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.814225 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2076347}
I20251024 19:48:29.815004 1507 raft_consensus.cc:2468] T 7deb97be82964ea0b2fe9caa867a6460 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:29.816248 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2671108031db449cae2135835fc5f739" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:29.816823 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7deb97be82964ea0b2fe9caa867a6460" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:29.817800 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet b95e41141f5f4a5bbc39fb977d82252b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b95e41141f5f4a5bbc39fb977d82252b already in progress: creating tablet
I20251024 19:48:29.818080 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet 182314f0f3074150b139cb2476976e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:29.818919 1586 leader_election.cc:343] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:29.819119 1586 leader_election.cc:343] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.819314 1586 leader_election.cc:304] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.820683 1768 tablet_bootstrap.cc:492] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:29.820722 3661 raft_consensus.cc:2804] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:29.820824 1768 ts_tablet_manager.cc:1403] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.000s
I20251024 19:48:29.820828 3661 raft_consensus.cc:697] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:29.821064 3661 consensus_queue.cc:237] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.821525 1768 raft_consensus.cc:359] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.821722 1768 raft_consensus.cc:385] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:29.821779 1768 raft_consensus.cc:740] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:29.821959 1768 consensus_queue.cc:260] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.822448 1768 ts_tablet_manager.cc:1434] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:29.822732 1768 tablet_bootstrap.cc:492] T b22b58404b8b441ba97ef536885a66e0 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:29.824240 1768 tablet_bootstrap.cc:654] T b22b58404b8b441ba97ef536885a66e0 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.825035 1774 tablet_replica.cc:333] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
W20251024 19:48:29.815507 1447 leader_election.cc:343] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.825177 1447 leader_election.cc:304] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:29.825285 1774 raft_consensus.cc:2243] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:29.825484 1774 pending_rounds.cc:70] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:29.825528 1774 pending_rounds.cc:77] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429425911525376 op_type: NO_OP noop_request { }
I20251024 19:48:29.825647 3760 raft_consensus.cc:2749] T 25ced70ff40d4149bb92e8b3767ec38f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:29.825671 1774 raft_consensus.cc:2889] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:29.825762 1774 raft_consensus.cc:2272] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:29.826197 1586 leader_election.cc:304] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.828195 3573 raft_consensus.cc:2804] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:29.828817 1774 ts_tablet_manager.cc:1916] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:29.830044 3573 raft_consensus.cc:697] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:29.830304 3573 consensus_queue.cc:237] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.838171 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 1b26130255b14a8f8b50f4eab86c20df (DEFAULT_TABLE table=default.Table_a0344b12f5f74a59a78491ee2968442a [id=9a3fa80693da49b791f2bd0ad1e729e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.838203 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 9035590b673740c2b81e11b55fe11601 (DEFAULT_TABLE table=default.Table_3fba2a24fecd460fa9482068292cc326 [id=45383554e9654a31b8bc3b35d642c9c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.838888 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 8d2783e12a854168bdcaff56265340f7 (DEFAULT_TABLE table=default.Table_795726c40abd4110a8d3173989dc70d5 [id=a6b6c19fd76e4f7abe788a0e68512068]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.838907 1628 tablet_service.cc:1505] Processing CreateTablet for tablet cd20e0fe863d4f29aacca76b3f68ed3d (DEFAULT_TABLE table=default.Table_42d562f8d96045689a6825f56d7f2c10 [id=09571109d2fa446f98248f2710a8093b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.839632 1625 tablet_service.cc:1505] Processing CreateTablet for tablet c773387f46b04e7e88a24c858b7c7f93 (DEFAULT_TABLE table=default.Table_c50cee70d21d437b917b616ea7cdddcb [id=effa6a3555194e1d857b465eaae4aca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.840015 1776 tablet_replica.cc:333] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:29.840245 1776 raft_consensus.cc:2243] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:29.840407 1776 raft_consensus.cc:2272] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:29.840979 1629 tablet_service.cc:1505] Processing CreateTablet for tablet f5d0f73b12264834b0e7d3622f4ce706 (DEFAULT_TABLE table=default.Table_e2fe881030b9494697581da39c43ebe2 [id=e16a3c87f43b431285c6a59d5c3d9dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.841120 1776 ts_tablet_manager.cc:1916] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:29.846127 1767 tablet_bootstrap.cc:492] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:29.848147 1767 tablet_bootstrap.cc:654] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:29.849082 1345 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ec69ffe2c024d1a8596a93183d1458b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.850679 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b26130255b14a8f8b50f4eab86c20df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.850911 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd20e0fe863d4f29aacca76b3f68ed3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.851302 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5d0f73b12264834b0e7d3622f4ce706. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.853519 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2783e12a854168bdcaff56265340f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.854050 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9035590b673740c2b81e11b55fe11601. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.854583 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c773387f46b04e7e88a24c858b7c7f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.855053 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be72a342153d4abbb3a29e19a9f0c82b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.855517 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de95bc9605e2423abf108f4be73cc8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.860106 1473 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ec69ffe2c024d1a8596a93183d1458b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:29.868502 1334 tablet_service.cc:1505] Processing CreateTablet for tablet cb269d7648f240f8b8b5827f0467e92f (DEFAULT_TABLE table=default.Table_db369541ede24bd69d04163bb5d48d62 [id=cc40491291f246aa93c24a4bb08f5f82]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:29.868927 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet a403201fab014ea98734c3e80649e088 (table default.Table_1ffe82b269b547ca938ef7eb9f4d8cc6 [id=877b65c08ba64fb798a9e137a17f4c4b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a403201fab014ea98734c3e80649e088 consensus state has no leader
I20251024 19:48:29.868989 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb269d7648f240f8b8b5827f0467e92f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.871541 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 182314f0f3074150b139cb2476976e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:29.872819 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 9035590b673740c2b81e11b55fe11601 (DEFAULT_TABLE table=default.Table_3fba2a24fecd460fa9482068292cc326 [id=45383554e9654a31b8bc3b35d642c9c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.873210 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9035590b673740c2b81e11b55fe11601. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.873291 1479 tablet_service.cc:1505] Processing CreateTablet for tablet 8d2783e12a854168bdcaff56265340f7 (DEFAULT_TABLE table=default.Table_795726c40abd4110a8d3173989dc70d5 [id=a6b6c19fd76e4f7abe788a0e68512068]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.873690 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2783e12a854168bdcaff56265340f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.878268 1489 tablet_service.cc:1505] Processing CreateTablet for tablet f5d0f73b12264834b0e7d3622f4ce706 (DEFAULT_TABLE table=default.Table_e2fe881030b9494697581da39c43ebe2 [id=e16a3c87f43b431285c6a59d5c3d9dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.878621 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5d0f73b12264834b0e7d3622f4ce706. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.881343 1339 tablet_service.cc:1505] Processing CreateTablet for tablet cd20e0fe863d4f29aacca76b3f68ed3d (DEFAULT_TABLE table=default.Table_42d562f8d96045689a6825f56d7f2c10 [id=09571109d2fa446f98248f2710a8093b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.881567 1192 master_service.cc:954] ReplaceTablet: received request to replace tablet 398e73d6d576487e968c9083175965aa from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:29.881724 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd20e0fe863d4f29aacca76b3f68ed3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.884085 1186 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_67e6d3d8a4334247a1bf237d5936799b" } modify_external_catalogs: true
I20251024 19:48:29.884235 1186 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_67e6d3d8a4334247a1bf237d5936799b" } modify_external_catalogs: true
I20251024 19:48:29.885600 1189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_3a9a22d932c84e0586bac3e1f92432e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.886081 1189 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a9a22d932c84e0586bac3e1f92432e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.886291 1350 tablet_service.cc:1505] Processing CreateTablet for tablet c773387f46b04e7e88a24c858b7c7f93 (DEFAULT_TABLE table=default.Table_c50cee70d21d437b917b616ea7cdddcb [id=effa6a3555194e1d857b465eaae4aca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.886613 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c773387f46b04e7e88a24c858b7c7f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.887243 1185 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1ffe82b269b547ca938ef7eb9f4d8cc6" } modify_external_catalogs: true
I20251024 19:48:29.887395 1185 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1ffe82b269b547ca938ef7eb9f4d8cc6" } modify_external_catalogs: true
I20251024 19:48:29.888588 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5f7d0b8e722a457ab176bb08207e178c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.889052 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f7d0b8e722a457ab176bb08207e178c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.889825 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 8d2783e12a854168bdcaff56265340f7 (DEFAULT_TABLE table=default.Table_795726c40abd4110a8d3173989dc70d5 [id=a6b6c19fd76e4f7abe788a0e68512068]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.889986 1336 tablet_service.cc:1505] Processing CreateTablet for tablet f5d0f73b12264834b0e7d3622f4ce706 (DEFAULT_TABLE table=default.Table_e2fe881030b9494697581da39c43ebe2 [id=e16a3c87f43b431285c6a59d5c3d9dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.889984 1485 tablet_service.cc:1505] Processing CreateTablet for tablet cd20e0fe863d4f29aacca76b3f68ed3d (DEFAULT_TABLE table=default.Table_42d562f8d96045689a6825f56d7f2c10 [id=09571109d2fa446f98248f2710a8093b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.890148 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2783e12a854168bdcaff56265340f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.890156 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_af70ad0ef8864f3a83c8376e84e7ddd3" } modify_external_catalogs: true
I20251024 19:48:29.890272 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_af70ad0ef8864f3a83c8376e84e7ddd3" } modify_external_catalogs: true
I20251024 19:48:29.890365 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd20e0fe863d4f29aacca76b3f68ed3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.890933 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5d0f73b12264834b0e7d3622f4ce706. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.893914 1475 tablet_service.cc:1505] Processing CreateTablet for tablet c773387f46b04e7e88a24c858b7c7f93 (DEFAULT_TABLE table=default.Table_c50cee70d21d437b917b616ea7cdddcb [id=effa6a3555194e1d857b465eaae4aca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.894246 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c773387f46b04e7e88a24c858b7c7f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.899740 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_46bec415c3904f1f943fcb1a24140e1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.900290 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46bec415c3904f1f943fcb1a24140e1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.900980 3760 raft_consensus.cc:493] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.901109 3760 raft_consensus.cc:515] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:29.901808 3760 leader_election.cc:290] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:29.901964 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_5bf3fd6ad2884dd8a4dcd772d793798d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.902408 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bf3fd6ad2884dd8a4dcd772d793798d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.902519 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be21ca442dee4204a8b15f5a2dd9530b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:29.903079 1444 leader_election.cc:343] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.906553 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "be21ca442dee4204a8b15f5a2dd9530b" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:29.910518 1447 leader_election.cc:343] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.910656 1447 leader_election.cc:304] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:29.911087 3760 raft_consensus.cc:2749] T be21ca442dee4204a8b15f5a2dd9530b P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:29.915169 3404 raft_consensus.cc:493] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.915304 3404 raft_consensus.cc:515] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.915877 3404 leader_election.cc:290] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.919107 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 1b26130255b14a8f8b50f4eab86c20df (DEFAULT_TABLE table=default.Table_a0344b12f5f74a59a78491ee2968442a [id=9a3fa80693da49b791f2bd0ad1e729e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.919544 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b26130255b14a8f8b50f4eab86c20df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.922958 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_bf5080a43b0841818ccdbb98b65e05ba" } new_table_name: "default.Table_d790da8cb00246108d3788c7cc05e326" modify_external_catalogs: true
W20251024 19:48:29.922991 1586 consensus_peers.cc:597] T 64499c8734d34c66b0b52a2cd3ef0673 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64499c8734d34c66b0b52a2cd3ef0673. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:29.923714 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f997c14171b3466f9c2c388c542735bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:29.924181 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f997c14171b3466f9c2c388c542735bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:29.924403 1367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774a2de2e3764c54b72acfb952ed3e11" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:29.924894 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774a2de2e3764c54b72acfb952ed3e11" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:29.925071 1507 raft_consensus.cc:2468] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:29.925945 1186 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet e6fd158e8b0c4cc78cf7fd3b34d5f104
W20251024 19:48:29.928409 1586 leader_election.cc:343] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:29.928931 1586 leader_election.cc:304] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:29.929175 3404 raft_consensus.cc:2804] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:29.929240 3404 raft_consensus.cc:493] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:29.929296 3404 raft_consensus.cc:3060] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.937012 3404 raft_consensus.cc:515] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.941931 1185 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet a403201fab014ea98734c3e80649e088
I20251024 19:48:29.944617 1775 ts_tablet_manager.cc:1929] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:29.945132 3404 leader_election.cc:290] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.939724 1625 tablet_service.cc:1505] Processing CreateTablet for tablet a43796bcdf7e433ca0241add9efb8483 (DEFAULT_TABLE table=default.Table_ae482796a941475d8e4597fa4a8a1264 [id=261c7c58bcd04fd69f2eadc422c57775]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.945655 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43796bcdf7e433ca0241add9efb8483. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.940294 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 93ac637f97494455baf2801b595674c3 (DEFAULT_TABLE table=default.Table_16bdfd2e16724b5a8baf4bf776eaa165 [id=c93d40037d504792b05b18ec305c333f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.951645 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93ac637f97494455baf2801b595674c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.940483 1611 tablet_service.cc:1552] Processing DeleteTablet for tablet e6fd158e8b0c4cc78cf7fd3b34d5f104 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.942822 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet a403201fab014ea98734c3e80649e088 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.956640 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet e6fd158e8b0c4cc78cf7fd3b34d5f104 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.958818 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 33afd3b5b43748d2a25880189a046a36 (DEFAULT_TABLE table=default.Table_23181a343624498bbcf28a5ab6b564eb [id=02dc4633885d4c19a59d6a25f0c647a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.959319 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33afd3b5b43748d2a25880189a046a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.966269 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774a2de2e3764c54b72acfb952ed3e11" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:29.967973 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 2b119e8ceae04852964513ebfa7aa31f
I20251024 19:48:29.968920 1612 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b119e8ceae04852964513ebfa7aa31f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:29.970249 1346 tablet_service.cc:1505] Processing CreateTablet for tablet a43796bcdf7e433ca0241add9efb8483 (DEFAULT_TABLE table=default.Table_ae482796a941475d8e4597fa4a8a1264 [id=261c7c58bcd04fd69f2eadc422c57775]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.970680 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43796bcdf7e433ca0241add9efb8483. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.950634 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet f88390f2534e4799a6eaa56aa4245e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:06 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:29.956332 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet e6fd158e8b0c4cc78cf7fd3b34d5f104 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:29.972187 1775 log.cc:1199] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/4a69c0918d2e4eed9d91b1913ec2c21e
I20251024 19:48:29.967790 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "774a2de2e3764c54b72acfb952ed3e11" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:29.966513 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 9035590b673740c2b81e11b55fe11601 (DEFAULT_TABLE table=default.Table_3fba2a24fecd460fa9482068292cc326 [id=45383554e9654a31b8bc3b35d642c9c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.972446 1507 raft_consensus.cc:3060] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:29.972803 1775 ts_tablet_manager.cc:1950] T 4a69c0918d2e4eed9d91b1913ec2c21e P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:29.972811 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9035590b673740c2b81e11b55fe11601. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.974977 1774 ts_tablet_manager.cc:1929] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:29.975448 1507 raft_consensus.cc:2468] T 774a2de2e3764c54b72acfb952ed3e11 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:29.977367 1776 ts_tablet_manager.cc:1929] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:29.977727 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 1b26130255b14a8f8b50f4eab86c20df (DEFAULT_TABLE table=default.Table_a0344b12f5f74a59a78491ee2968442a [id=9a3fa80693da49b791f2bd0ad1e729e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.978173 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b26130255b14a8f8b50f4eab86c20df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.981505 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 93ac637f97494455baf2801b595674c3 (DEFAULT_TABLE table=default.Table_16bdfd2e16724b5a8baf4bf776eaa165 [id=c93d40037d504792b05b18ec305c333f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.981899 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93ac637f97494455baf2801b595674c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.976394 1337 tablet_service.cc:1505] Processing CreateTablet for tablet de95bc9605e2423abf108f4be73cc8b8 (DEFAULT_TABLE table=default.Table_aadd52f1943441bda187b984246fcc02 [id=62e3c1cde81741718077889e71dab17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.982542 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de95bc9605e2423abf108f4be73cc8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.985052 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 1286dac5e8d24d50a652d3a485814159 (DEFAULT_TABLE table=default.Table_cc788c9fa46e4cdb9fcfe197121d4253 [id=08c466bccc854f82a9da4212e7ce813f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.985416 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1286dac5e8d24d50a652d3a485814159. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.977250 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet 182314f0f3074150b139cb2476976e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.980942 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 93ac637f97494455baf2801b595674c3 (DEFAULT_TABLE table=default.Table_16bdfd2e16724b5a8baf4bf776eaa165 [id=c93d40037d504792b05b18ec305c333f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.985420 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 33afd3b5b43748d2a25880189a046a36 (DEFAULT_TABLE table=default.Table_23181a343624498bbcf28a5ab6b564eb [id=02dc4633885d4c19a59d6a25f0c647a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.985947 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93ac637f97494455baf2801b595674c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.985961 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33afd3b5b43748d2a25880189a046a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.988569 3787 raft_consensus.cc:493] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:29.988687 3787 raft_consensus.cc:515] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:29.989300 3787 leader_election.cc:290] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:29.995537 1774 log.cc:1199] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/ab488e4971874a16b2e052c84c250777
I20251024 19:48:29.995990 1774 ts_tablet_manager.cc:1950] T ab488e4971874a16b2e052c84c250777 P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:29.998008 1475 tablet_service.cc:1552] Processing DeleteTablet for tablet a403201fab014ea98734c3e80649e088 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:29.998589 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1209) took 4623 ms. Trace:
I20251024 19:48:29.998734 1775 rpcz_store.cc:276] 1024 19:48:25.375057 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.375118 (+ 61us) service_pool.cc:225] Handling call
1024 19:48:29.688335 (+4313217us) ts_tablet_manager.cc:1151] Deleting tablet 4a69c0918d2e4eed9d91b1913ec2c21e
1024 19:48:29.688345 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.866249 (+177904us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:29.944488 (+ 78239us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:29.998567 (+ 54079us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4313010}
I20251024 19:48:29.999306 1775 tablet_replica.cc:333] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:29.999727 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 33afd3b5b43748d2a25880189a046a36 (DEFAULT_TABLE table=default.Table_23181a343624498bbcf28a5ab6b564eb [id=02dc4633885d4c19a59d6a25f0c647a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.000192 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33afd3b5b43748d2a25880189a046a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.001960 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 4a69c0918d2e4eed9d91b1913ec2c21e (table default.Table_8b8ee145388d4588b50306be38739650 [id=8f925cc7435543aa8aacba668bd66b52]) successfully deleted
I20251024 19:48:30.002277 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet a403201fab014ea98734c3e80649e088 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:29.990612 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebcc09d8e029451d8fb6b6601ad16087" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:30.002429 1776 log.cc:1199] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/80c58e2439dc490c824b5f0023b5434b
I20251024 19:48:30.002547 1646 raft_consensus.cc:2393] T ebcc09d8e029451d8fb6b6601ad16087 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate 10a9536207e343f188e0ef709081d04e in this term.
I20251024 19:48:30.002784 1776 ts_tablet_manager.cc:1950] T 80c58e2439dc490c824b5f0023b5434b P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:29.997563 1612 tablet_service.cc:1505] Processing CreateTablet for tablet 1286dac5e8d24d50a652d3a485814159 (DEFAULT_TABLE table=default.Table_cc788c9fa46e4cdb9fcfe197121d4253 [id=08c466bccc854f82a9da4212e7ce813f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:29.998101 1611 tablet_service.cc:1505] Processing CreateTablet for tablet c50dca825b0d4b0c84f6049c74e617cd (DEFAULT_TABLE table=default.Table_2f474456e9334431b6a1d56c9f4c911b [id=976524d640e04141b67ff3624a0c47d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.001081 1473 tablet_service.cc:1505] Processing CreateTablet for tablet a43796bcdf7e433ca0241add9efb8483 (DEFAULT_TABLE table=default.Table_ae482796a941475d8e4597fa4a8a1264 [id=261c7c58bcd04fd69f2eadc422c57775]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.005544 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43796bcdf7e433ca0241add9efb8483. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.009359 1775 raft_consensus.cc:2243] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:30.009752 1775 raft_consensus.cc:2272] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:30.010648 1774 tablet_replica.cc:333] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:30.017118 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1295) took 2282 ms. Trace:
I20251024 19:48:30.017318 1776 rpcz_store.cc:276] 1024 19:48:27.734382 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.734441 (+ 59us) service_pool.cc:225] Handling call
1024 19:48:29.814789 (+2080348us) ts_tablet_manager.cc:1151] Deleting tablet 80c58e2439dc490c824b5f0023b5434b
1024 19:48:29.839639 (+ 24850us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:29.871477 (+ 31838us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:29.977254 (+105777us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:30.017087 (+ 39833us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1408,"tablet-delete.queue_time_us":2080158}
I20251024 19:48:30.017906 1192 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 398e73d6d576487e968c9083175965aa
I20251024 19:48:30.018101 1776 tablet_replica.cc:333] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:30.018299 1776 raft_consensus.cc:2243] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:30.018364 1192 catalog_manager.cc:6719] ReplaceTablet: tablet 398e73d6d576487e968c9083175965aa deleted and replaced by tablet e9384e55729c4d5eb073c754a172a52a
I20251024 19:48:30.018714 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50dca825b0d4b0c84f6049c74e617cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.018767 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 80c58e2439dc490c824b5f0023b5434b (table default.Table_2aa9405152c14d538c7164b50b336c7f [id=c00489e309ba4ada921f101a95bc8b48]) successfully deleted
I20251024 19:48:30.019037 1630 tablet_service.cc:1552] Processing DeleteTablet for tablet 398e73d6d576487e968c9083175965aa with delete_type TABLET_DATA_DELETED (replaced by tablet e9384e55729c4d5eb073c754a172a52a) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:30.021806 3787 raft_consensus.cc:493] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:30.021948 3787 raft_consensus.cc:515] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:30.022055 1630 tablet_service.cc:1505] Processing CreateTablet for tablet 9ffc598d59674ec2bcfcd8b656cc533f (DEFAULT_TABLE table=default.Table_37671d233b204f0f81b63e1518018725 [id=f2f912a6c50e4fcdad32d3b096edbf97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.022580 3787 leader_election.cc:290] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045), af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119)
I20251024 19:48:30.023077 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ffc598d59674ec2bcfcd8b656cc533f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.025738 1646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6f8764e9ae472b9ac35a032182aed1" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:30.028133 1775 ts_tablet_manager.cc:1916] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:30.027908 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet ab488e4971874a16b2e052c84c250777 (table default.Table_93cc576cc317428c88108b05bf8efc6d [id=c69469ba6a9745f6a66092a678a5861d]) successfully deleted
W20251024 19:48:30.029821 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4438d3865f3c41d694428f31db6e4da7 consensus state has no leader
I20251024 19:48:30.030020 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1286dac5e8d24d50a652d3a485814159. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.031143 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7060116dc4504d2fb09b13668396d97a" } new_table_name: "default.Table_9560ca189e024802934c9462f1671971" modify_external_catalogs: true
I20251024 19:48:30.032315 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e6f8764e9ae472b9ac35a032182aed1" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:30.032567 1508 raft_consensus.cc:2393] T 3e6f8764e9ae472b9ac35a032182aed1 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43bec2f03fc849099e254f6335e9d532 in current term 1: Already voted for candidate af3952de050140d383f6b0de7666fa97 in this term.
I20251024 19:48:30.032436 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4102ac21c0a345da986f9f8f4d66a1b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:30.032940 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4102ac21c0a345da986f9f8f4d66a1b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:30.032986 1308 leader_election.cc:304] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:30.033396 3787 raft_consensus.cc:3060] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:30.034776 1629 tablet_service.cc:1505] Processing CreateTablet for tablet c9e13cb746614bef9ad01fe45ea918d1 (DEFAULT_TABLE table=default.Table_f4dae23457914d48b92be0b74929c89b [id=ddf4570360e24d7591cbebe94417e58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.035120 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e13cb746614bef9ad01fe45ea918d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.035974 1183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_fa48e693d40446bdbc062d650fadfc0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:30.036334 3787 raft_consensus.cc:2749] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251024 19:48:30.036391 1183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa48e693d40446bdbc062d650fadfc0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:30.038235 1630 tablet_service.cc:1505] Processing CreateTablet for tablet b60bbdb685af430e8042ffd78b99869b (DEFAULT_TABLE table=default.Table_07368d3d5ebf4c10baed00c0c48d9769 [id=1a6068b763e24ab89014987604309620]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.038457 1624 tablet_service.cc:1505] Processing CreateTablet for tablet eb9801bfcb5c42878ecf1b4d3605fce0 (DEFAULT_TABLE table=default.Table_f75802f81eff41248b27d50a04c4efbc [id=cc417c0457034d709f36cbac1f56179c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.038599 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b60bbdb685af430e8042ffd78b99869b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.038802 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb9801bfcb5c42878ecf1b4d3605fce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.038472 1628 tablet_service.cc:1505] Processing CreateTablet for tablet d6e804268f584f8fbde5e1e8719b8189 (DEFAULT_TABLE table=default.Table_d078179b97b04326aefe7ca93fdc335b [id=96bda312f6e84807a893de7959d6daa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.039041 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6e804268f584f8fbde5e1e8719b8189. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:29.815577 1766 tablet_bootstrap.cc:492] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:30.044670 1766 ts_tablet_manager.cc:1403] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.390s user 0.003s sys 0.001s
I20251024 19:48:30.045403 1766 raft_consensus.cc:359] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:30.046106 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebcc09d8e029451d8fb6b6601ad16087" candidate_uuid: "43bec2f03fc849099e254f6335e9d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:30.046641 1308 leader_election.cc:304] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43bec2f03fc849099e254f6335e9d532; no voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97
I20251024 19:48:30.046814 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_54069a0bafeb429690cbf406ac98eabe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:30.047231 3787 raft_consensus.cc:3060] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:30.047475 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54069a0bafeb429690cbf406ac98eabe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:30.049751 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b119e8ceae04852964513ebfa7aa31f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:30.051373 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b119e8ceae04852964513ebfa7aa31f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:29 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:30.052254 3787 raft_consensus.cc:2749] T ebcc09d8e029451d8fb6b6601ad16087 P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:30.054250 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 1286dac5e8d24d50a652d3a485814159 (DEFAULT_TABLE table=default.Table_cc788c9fa46e4cdb9fcfe197121d4253 [id=08c466bccc854f82a9da4212e7ce813f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.054639 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1286dac5e8d24d50a652d3a485814159. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:30.064694 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57e1e21beee44847b1ebeb746f02c372 consensus state has no leader
I20251024 19:48:30.068913 1485 tablet_service.cc:1505] Processing CreateTablet for tablet c50dca825b0d4b0c84f6049c74e617cd (DEFAULT_TABLE table=default.Table_2f474456e9334431b6a1d56c9f4c911b [id=976524d640e04141b67ff3624a0c47d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.069278 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50dca825b0d4b0c84f6049c74e617cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.071101 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 398e73d6d576487e968c9083175965aa with delete_type TABLET_DATA_DELETED (replaced by tablet e9384e55729c4d5eb073c754a172a52a) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:30.073421 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 9ffc598d59674ec2bcfcd8b656cc533f (DEFAULT_TABLE table=default.Table_37671d233b204f0f81b63e1518018725 [id=f2f912a6c50e4fcdad32d3b096edbf97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.073832 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ffc598d59674ec2bcfcd8b656cc533f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.074184 1766 raft_consensus.cc:385] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:30.074311 1766 raft_consensus.cc:740] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:30.074651 1766 consensus_queue.cc:260] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:30.075449 1766 ts_tablet_manager.cc:1434] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20251024 19:48:30.075770 1766 tablet_bootstrap.cc:492] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:30.077183 1766 tablet_bootstrap.cc:654] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:30.077404 1475 tablet_service.cc:1505] Processing CreateTablet for tablet c9e13cb746614bef9ad01fe45ea918d1 (DEFAULT_TABLE table=default.Table_f4dae23457914d48b92be0b74929c89b [id=ddf4570360e24d7591cbebe94417e58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.077783 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e13cb746614bef9ad01fe45ea918d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.078780 1480 tablet_service.cc:1505] Processing CreateTablet for tablet b60bbdb685af430e8042ffd78b99869b (DEFAULT_TABLE table=default.Table_07368d3d5ebf4c10baed00c0c48d9769 [id=1a6068b763e24ab89014987604309620]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.079093 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b60bbdb685af430e8042ffd78b99869b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.083794 1489 tablet_service.cc:1505] Processing CreateTablet for tablet d6e804268f584f8fbde5e1e8719b8189 (DEFAULT_TABLE table=default.Table_d078179b97b04326aefe7ca93fdc335b [id=96bda312f6e84807a893de7959d6daa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.084236 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6e804268f584f8fbde5e1e8719b8189. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:30.086068 1016 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 47521.8us
I20251024 19:48:30.090689 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 9ffc598d59674ec2bcfcd8b656cc533f (DEFAULT_TABLE table=default.Table_37671d233b204f0f81b63e1518018725 [id=f2f912a6c50e4fcdad32d3b096edbf97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.091102 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ffc598d59674ec2bcfcd8b656cc533f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.091176 1335 tablet_service.cc:1505] Processing CreateTablet for tablet f5848ceed000455f96a199b5f9a70969 (DEFAULT_TABLE table=default.Table_f7f2e6d903ed4b139eaae42e0dc95366 [id=3d9bfe7a3a90415980fc639c42d38e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.091185 1350 tablet_service.cc:1505] Processing CreateTablet for tablet eb9801bfcb5c42878ecf1b4d3605fce0 (DEFAULT_TABLE table=default.Table_f75802f81eff41248b27d50a04c4efbc [id=cc417c0457034d709f36cbac1f56179c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.091487 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5848ceed000455f96a199b5f9a70969. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.091637 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb9801bfcb5c42878ecf1b4d3605fce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.093721 1338 tablet_service.cc:1505] Processing CreateTablet for tablet d6e804268f584f8fbde5e1e8719b8189 (DEFAULT_TABLE table=default.Table_d078179b97b04326aefe7ca93fdc335b [id=96bda312f6e84807a893de7959d6daa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.097812 1336 tablet_service.cc:1505] Processing CreateTablet for tablet b60bbdb685af430e8042ffd78b99869b (DEFAULT_TABLE table=default.Table_07368d3d5ebf4c10baed00c0c48d9769 [id=1a6068b763e24ab89014987604309620]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.098251 1629 tablet_service.cc:1505] Processing CreateTablet for tablet 40db8869778a46a0b8e26847d19753a6 (DEFAULT_TABLE table=default.Table_f1b82a4bcb2d4cca842b1be702afda63 [id=bb53c691d85f471bb66c71780f9c93e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.098628 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40db8869778a46a0b8e26847d19753a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.098697 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6e804268f584f8fbde5e1e8719b8189. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.099658 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 93f2efbb052847419ae097b4bdc92fe1 (DEFAULT_TABLE table=default.Table_d61d046099f34d0284dcee14b932fc2b [id=12ef333efb4f45709c131bf13b045ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.099969 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93f2efbb052847419ae097b4bdc92fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.101814 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 505edb8d2fd44c83badef6d36ea0abd9 (DEFAULT_TABLE table=default.Table_d257560a020c4081a4f874b3731b7921 [id=b394223de7894c70b7aee0d2828271d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.102113 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505edb8d2fd44c83badef6d36ea0abd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.106797 1775 ts_tablet_manager.cc:1929] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:30.112352 1339 tablet_service.cc:1505] Processing CreateTablet for tablet c9e13cb746614bef9ad01fe45ea918d1 (DEFAULT_TABLE table=default.Table_f4dae23457914d48b92be0b74929c89b [id=ddf4570360e24d7591cbebe94417e58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.117425 1627 tablet_service.cc:1505] Processing CreateTablet for tablet f03048deb8b94b80924375bf3db068e9 (DEFAULT_TABLE table=default.Table_1175a8d8c6144a8d9479b68f16c35b05 [id=2131ad6aa3f040228db72a3cbe0e8bf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.117880 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f03048deb8b94b80924375bf3db068e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.119069 1475 tablet_service.cc:1505] Processing CreateTablet for tablet eb9801bfcb5c42878ecf1b4d3605fce0 (DEFAULT_TABLE table=default.Table_f75802f81eff41248b27d50a04c4efbc [id=cc417c0457034d709f36cbac1f56179c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.119187 1775 log.cc:1199] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/e38ef2ac19d14f9c91fa8abe86cc7987
I20251024 19:48:30.119601 1775 ts_tablet_manager.cc:1950] T e38ef2ac19d14f9c91fa8abe86cc7987 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:30.121718 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb9801bfcb5c42878ecf1b4d3605fce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.124042 1767 tablet_bootstrap.cc:492] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:30.124156 1767 ts_tablet_manager.cc:1403] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.278s user 0.002s sys 0.013s
I20251024 19:48:30.124874 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 551479125efa4d58afb788651fa19c48 (DEFAULT_TABLE table=default.Table_5f5eee56374d49cd8a80b28f26f42bac [id=a3404453f85340068b15206987ffdc0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.125028 1767 raft_consensus.cc:359] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:30.125212 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 551479125efa4d58afb788651fa19c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.125221 1767 raft_consensus.cc:385] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:30.126154 1767 raft_consensus.cc:740] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:30.126363 1767 consensus_queue.cc:260] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:30.130038 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 4f1d4542e7df4473a19f75f6037398b9 (DEFAULT_TABLE table=default.Table_5035151c0890403b9dc59e3973c94e76 [id=69f8936a0e804a989dda3c35852f7666]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.130438 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1d4542e7df4473a19f75f6037398b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.099298 1346 tablet_service.cc:1552] Processing DeleteTablet for tablet 398e73d6d576487e968c9083175965aa with delete_type TABLET_DATA_DELETED (replaced by tablet e9384e55729c4d5eb073c754a172a52a) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:30.135452 1616 tablet_service.cc:1505] Processing CreateTablet for tablet daf07ab1ef05424b94c0611276979639 (DEFAULT_TABLE table=default.Table_286aa7130af04dcfb71514e1f9add870 [id=8a6be145b6d9418b8b081c173326f01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.135848 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf07ab1ef05424b94c0611276979639. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.136096 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 8efbe6b9e59b4e91916bb6a3ea6f4c60 (DEFAULT_TABLE table=default.Table_d3cbf85ffa894f149214169d2e8c9977 [id=f1bfde166821406eab8d91dc94bad2ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.136400 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8efbe6b9e59b4e91916bb6a3ea6f4c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.139118 3679 consensus_queue.cc:1048] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:30.140126 1767 ts_tablet_manager.cc:1434] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.016s user 0.001s sys 0.005s
I20251024 19:48:30.142889 1626 tablet_service.cc:1505] Processing CreateTablet for tablet a54c6469dcea4b6da6d23c8395482c6f (DEFAULT_TABLE table=default.Table_34787bfb5ad443ff9c6ef9a773ddb6ff [id=d862b8cb002f407fab74b378d2501285]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.143241 1626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a54c6469dcea4b6da6d23c8395482c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.150472 1621 tablet_service.cc:1505] Processing CreateTablet for tablet aea6f26c2b5144e78f3332ef67b14bb9 (DEFAULT_TABLE table=default.Table_09fd5426c34f40bc9ae783f13695e8bc [id=37625ba1414f4a1ebafa0561bd2e2058]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.150895 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aea6f26c2b5144e78f3332ef67b14bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.154204 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 61b4135223a747198fcb4a6f052f4f08 (DEFAULT_TABLE table=default.Table_2d40dad65a34427ca08b38d576c386aa [id=545f5ceff0b44feea46b75496958d657]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.154594 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b4135223a747198fcb4a6f052f4f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.158216 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 47c9fcf87a194c08a4e1de84dfc96f42 (DEFAULT_TABLE table=default.Table_a67503cccddd4a6e9d8ea3b30488cc14 [id=3b5b8389d9bf4203a0d2844ebac40304]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.158555 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c9fcf87a194c08a4e1de84dfc96f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.162379 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 20b6f6453f4d438a96e30fb30b20a8bd (DEFAULT_TABLE table=default.Table_42d61eaa29c64c7281f5cb3cf361b9e1 [id=d8d60aac53334cb3a71b27a010001481]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.163304 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20b6f6453f4d438a96e30fb30b20a8bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:30.163378 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 195859us
I20251024 19:48:30.164906 1776 pending_rounds.cc:70] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:30.164956 1776 pending_rounds.cc:77] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429419118616576 op_type: NO_OP noop_request { }
I20251024 19:48:30.165055 1776 raft_consensus.cc:2889] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:30.165167 1776 raft_consensus.cc:2272] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:30.165771 1776 ts_tablet_manager.cc:1916] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:30.172710 1776 ts_tablet_manager.cc:1929] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:30.173166 1776 log.cc:1199] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/57f522a825264ea7b215304dbb7aa2d5
I20251024 19:48:30.173563 1776 ts_tablet_manager.cc:1950] T 57f522a825264ea7b215304dbb7aa2d5 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:30.173659 3800 tablet.cc:1722] T e0d5d210d8d8499cbfdbdaa29e5bb40b P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:30.182353 3404 raft_consensus.cc:493] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:30.182489 3404 raft_consensus.cc:515] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.183185 3404 leader_election.cc:290] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:30.186030 1489 tablet_service.cc:1505] Processing CreateTablet for tablet f5848ceed000455f96a199b5f9a70969 (DEFAULT_TABLE table=default.Table_f7f2e6d903ed4b139eaae42e0dc95366 [id=3d9bfe7a3a90415980fc639c42d38e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.186625 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 40db8869778a46a0b8e26847d19753a6 (DEFAULT_TABLE table=default.Table_f1b82a4bcb2d4cca842b1be702afda63 [id=bb53c691d85f471bb66c71780f9c93e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.187011 1480 tablet_service.cc:1505] Processing CreateTablet for tablet 93f2efbb052847419ae097b4bdc92fe1 (DEFAULT_TABLE table=default.Table_d61d046099f34d0284dcee14b932fc2b [id=12ef333efb4f45709c131bf13b045ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.187634 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 505edb8d2fd44c83badef6d36ea0abd9 (DEFAULT_TABLE table=default.Table_d257560a020c4081a4f874b3731b7921 [id=b394223de7894c70b7aee0d2828271d4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:30.188120 1586 leader_election.cc:343] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:30.188362 1586 leader_election.cc:304] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:30.188661 3404 raft_consensus.cc:2804] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:30.188815 3404 raft_consensus.cc:697] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:30.189179 3404 consensus_queue.cc:237] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.200160 3825 raft_consensus.cc:493] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:30.200285 3825 raft_consensus.cc:515] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:30.200824 3825 leader_election.cc:290] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:30.201294 3800 tablet.cc:1722] T c512ec26e3874cd7869216b1fb90fa54 P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:30.203518 1192 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_1871aba06fbd431a8cebf906fbd2f854 [id=a2adb68f20424f9a8b8d8785496b2f3f] alter complete (version 1)
I20251024 19:48:30.203639 1613 tablet_service.cc:1505] Processing CreateTablet for tablet f5848ceed000455f96a199b5f9a70969 (DEFAULT_TABLE table=default.Table_f7f2e6d903ed4b139eaae42e0dc95366 [id=3d9bfe7a3a90415980fc639c42d38e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.204015 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5848ceed000455f96a199b5f9a70969. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.205854 1337 tablet_service.cc:1505] Processing CreateTablet for tablet c50dca825b0d4b0c84f6049c74e617cd (DEFAULT_TABLE table=default.Table_2f474456e9334431b6a1d56c9f4c911b [id=976524d640e04141b67ff3624a0c47d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.207486 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 40db8869778a46a0b8e26847d19753a6 (DEFAULT_TABLE table=default.Table_f1b82a4bcb2d4cca842b1be702afda63 [id=bb53c691d85f471bb66c71780f9c93e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.214684 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15de0d8aa9e545c09233186bbe361b97" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:30.214848 1766 tablet_bootstrap.cc:492] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:30.215015 1766 ts_tablet_manager.cc:1403] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.000s
I20251024 19:48:30.215801 1766 raft_consensus.cc:359] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:30.216068 1766 raft_consensus.cc:385] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:30.216177 1766 raft_consensus.cc:740] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:30.216596 1766 consensus_queue.cc:260] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:30.216846 1183 catalog_manager.cc:5649] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:30.217454 1183 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 2671108031db449cae2135835fc5f739 (table default.Table_8514e876be154443ad1f2827119421c5 [id=2f6b7b685af84337bd64141ef2a9e992]). Expected version 3 got 0
I20251024 19:48:30.217550 1766 ts_tablet_manager.cc:1434] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251024 19:48:30.218600 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b60bbdb685af430e8042ffd78b99869b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.219614 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e13cb746614bef9ad01fe45ea918d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.220124 1509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15de0d8aa9e545c09233186bbe361b97" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:30.220329 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40db8869778a46a0b8e26847d19753a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.220366 1509 raft_consensus.cc:2468] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:30.220574 1505 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b153f75caebb4d869e77529c9c9bb50d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:30.220727 1505 raft_consensus.cc:2468] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:30.221184 3804 tablet.cc:1722] T c512ec26e3874cd7869216b1fb90fa54 P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:30.221797 3679 raft_consensus.cc:493] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:30.221954 3679 raft_consensus.cc:515] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:30.222668 3679 leader_election.cc:290] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:30.225179 1645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "163ac050c8574615aef5061f563cfc0a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:30.225807 1444 leader_election.cc:343] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:30.226632 1629 tablet_service.cc:1505] Processing CreateTablet for tablet d1ac612a1a80485ea3ebb8e6e4e8e26e (DEFAULT_TABLE table=default.Table_312699096d58411cbc14672a7c9c7d9f [id=68f1bca8d63840529238be11ba2bbfce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.227041 1629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1ac612a1a80485ea3ebb8e6e4e8e26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.227958 1624 tablet_service.cc:1505] Processing CreateTablet for tablet e7a14fb9816f4120bc31caab65bd8273 (DEFAULT_TABLE table=default.Table_db1ad1d0cb0f4bfd9cb4fc16270c74d3 [id=447b25ec36984cbfa4c5200129daddd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.228467 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7a14fb9816f4120bc31caab65bd8273. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.230334 1625 tablet_service.cc:1505] Processing CreateTablet for tablet f56f3d0e9e0a41ce91c0eeabe0b6dda1 (DEFAULT_TABLE table=default.Table_435492c6b1164482acacb1acbaa70afd [id=e2e000507ac74d958cd3c036cd90cc56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.230612 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40db8869778a46a0b8e26847d19753a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.230657 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f56f3d0e9e0a41ce91c0eeabe0b6dda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.231796 1619 tablet_service.cc:1505] Processing CreateTablet for tablet 25645023adfc4adc838a27d78bc9a94c (DEFAULT_TABLE table=default.Table_21b929f19b444807ba23a7a062df8e34 [id=0b86fa21ecae403fa2fead4ba221dc0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.232185 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25645023adfc4adc838a27d78bc9a94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.233748 1767 tablet_bootstrap.cc:492] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:30.236897 1767 tablet_bootstrap.cc:654] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:30.239949 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1294) took 2505 ms. Trace:
I20251024 19:48:30.240041 1776 rpcz_store.cc:276] 1024 19:48:27.734207 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.734659 (+ 452us) service_pool.cc:225] Handling call
1024 19:48:30.018045 (+2283386us) ts_tablet_manager.cc:1151] Deleting tablet 57f522a825264ea7b215304dbb7aa2d5
1024 19:48:30.018056 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.164649 (+146593us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x6170001331f0. stack: 00007fea3c4169f6 00007fea3c4167b0 00007fea3b3f23ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fea481bb76f 00007fea4826bf5c 00007fea482dc1e3 00007fea49629ae3 00007fea4c1e6287 00007fea4c2192d4 00007fea4c1f9d70 00007fea4c1f9afd 0000000000a8867c 00007fea3c45cd7f
1024 19:48:30.169424 (+ 4775us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:30.172626 (+ 3202us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:30.239932 (+ 67306us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63380,"spinlock_wait_cycles":334646144,"tablet-delete.queue_time_us":2283244}
I20251024 19:48:30.240479 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1296) took 2495 ms. Trace:
I20251024 19:48:30.240558 1776 rpcz_store.cc:276] 1024 19:48:27.744963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.745043 (+ 80us) service_pool.cc:225] Handling call
1024 19:48:30.240361 (+2495318us) ts_tablet_manager.cc:1151] Deleting tablet f405fc8974e243af982f30e00607d14a
1024 19:48:30.240370 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.240462 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2495124}
I20251024 19:48:30.240877 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1297) took 2481 ms. Trace:
I20251024 19:48:30.240945 1776 rpcz_store.cc:276] 1024 19:48:27.759917 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.759980 (+ 63us) service_pool.cc:225] Handling call
1024 19:48:30.240779 (+2480799us) ts_tablet_manager.cc:1151] Deleting tablet be21ca442dee4204a8b15f5a2dd9530b
1024 19:48:30.240787 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.240861 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2480567}
I20251024 19:48:30.241176 1776 tablet_replica.cc:333] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:30.241364 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1210) took 4865 ms. Trace:
I20251024 19:48:30.241387 1776 raft_consensus.cc:2243] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:30.241458 1775 rpcz_store.cc:276] 1024 19:48:25.375552 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.375596 (+ 44us) service_pool.cc:225] Handling call
1024 19:48:29.999262 (+4623666us) ts_tablet_manager.cc:1151] Deleting tablet e38ef2ac19d14f9c91fa8abe86cc7987
1024 19:48:29.999271 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.059682 (+ 60411us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:30.106578 (+ 46896us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:30.241336 (+134758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4623521}
I20251024 19:48:30.242048 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1219) took 4789 ms. Trace:
I20251024 19:48:30.242165 1775 rpcz_store.cc:276] 1024 19:48:25.452812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.452865 (+ 53us) service_pool.cc:225] Handling call
1024 19:48:30.241893 (+4789028us) ts_tablet_manager.cc:1151] Deleting tablet 28f93be6762146398421ac19d2e84e0c
1024 19:48:30.241904 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.242029 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4788835}
I20251024 19:48:30.242352 1776 pending_rounds.cc:70] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97: Trying to abort 1 pending ops.
I20251024 19:48:30.242413 1776 pending_rounds.cc:77] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429423151026176 op_type: NO_OP noop_request { }
I20251024 19:48:30.242529 1776 raft_consensus.cc:2889] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:30.242611 1776 raft_consensus.cc:2272] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:30.242683 1775 tablet_replica.cc:333] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:30.243001 1775 raft_consensus.cc:2243] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:30.243167 1775 raft_consensus.cc:2272] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:30.243188 1776 ts_tablet_manager.cc:1916] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:30.243736 1775 ts_tablet_manager.cc:1916] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:30.245055 1586 consensus_peers.cc:597] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 -> Peer 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): Couldn't send request to peer 10a9536207e343f188e0ef709081d04e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a69c0918d2e4eed9d91b1913ec2c21e. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:30.245416 1586 leader_election.cc:304] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:30.245731 3573 raft_consensus.cc:2804] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:30.245819 3573 raft_consensus.cc:493] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:30.245893 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505edb8d2fd44c83badef6d36ea0abd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.245890 3573 raft_consensus.cc:3060] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:30.249182 3573 raft_consensus.cc:515] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.249855 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5848ceed000455f96a199b5f9a70969. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.250140 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93f2efbb052847419ae097b4bdc92fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.250778 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 57f522a825264ea7b215304dbb7aa2d5 (table default.Table_6a4dbd8e2b264bbca433946dfb1890b7 [id=9fec2fa31d474c5f8bc3b39041ed30bb]) successfully deleted
W20251024 19:48:30.251238 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f405fc8974e243af982f30e00607d14a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f405fc8974e243af982f30e00607d14a already in progress: creating tablet
W20251024 19:48:30.251447 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet be21ca442dee4204a8b15f5a2dd9530b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be21ca442dee4204a8b15f5a2dd9530b already in progress: creating tablet
I20251024 19:48:30.252871 1776 ts_tablet_manager.cc:1929] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:30.253105 1776 log.cc:1199] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/83b91a5d764449d5b82ef6d1ee0360f5
I20251024 19:48:30.253886 1776 ts_tablet_manager.cc:1950] T 83b91a5d764449d5b82ef6d1ee0360f5 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:30.254557 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1298) took 2431 ms. Trace:
I20251024 19:48:30.254760 1776 rpcz_store.cc:276] 1024 19:48:27.823348 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:27.823420 (+ 72us) service_pool.cc:225] Handling call
1024 19:48:30.241134 (+2417714us) ts_tablet_manager.cc:1151] Deleting tablet 83b91a5d764449d5b82ef6d1ee0360f5
1024 19:48:30.241143 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.247337 (+ 6194us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:30.252761 (+ 5424us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:30.254510 (+ 1749us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2417456}
I20251024 19:48:30.255491 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 83b91a5d764449d5b82ef6d1ee0360f5 (table default.Table_325e96b0d32e4a3aab388a916c29ec26 [id=c6b5c56913684d08b89fde458e31b8cb]) successfully deleted
I20251024 19:48:30.265236 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50dca825b0d4b0c84f6049c74e617cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.266158 1586 leader_election.cc:304] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:30.215751 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 505edb8d2fd44c83badef6d36ea0abd9 (DEFAULT_TABLE table=default.Table_d257560a020c4081a4f874b3731b7921 [id=b394223de7894c70b7aee0d2828271d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.266531 3738 raft_consensus.cc:2804] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:30.266686 3738 raft_consensus.cc:493] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:30.266767 3738 raft_consensus.cc:3060] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:30.266723 3825 consensus_queue.cc:1048] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:30.266842 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505edb8d2fd44c83badef6d36ea0abd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.267381 3825 consensus_queue.cc:1048] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:30.268404 1342 tablet_service.cc:1505] Processing CreateTablet for tablet f03048deb8b94b80924375bf3db068e9 (DEFAULT_TABLE table=default.Table_1175a8d8c6144a8d9479b68f16c35b05 [id=2131ad6aa3f040228db72a3cbe0e8bf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.268759 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f03048deb8b94b80924375bf3db068e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.269443 3738 raft_consensus.cc:515] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:30.270355 3738 leader_election.cc:290] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:30.270689 3679 consensus_queue.cc:1048] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:30.273337 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b153f75caebb4d869e77529c9c9bb50d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:30.275648 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 93f2efbb052847419ae097b4bdc92fe1 (DEFAULT_TABLE table=default.Table_d61d046099f34d0284dcee14b932fc2b [id=12ef333efb4f45709c131bf13b045ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.276247 3573 leader_election.cc:290] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:30.277089 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "163ac050c8574615aef5061f563cfc0a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:30.279131 1447 leader_election.cc:343] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:30.279258 1447 leader_election.cc:304] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:30.279824 3759 raft_consensus.cc:2749] T 163ac050c8574615aef5061f563cfc0a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:30.280184 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93f2efbb052847419ae097b4bdc92fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.286616 1767 tablet_bootstrap.cc:492] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:30.286793 1767 ts_tablet_manager.cc:1403] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.006s
I20251024 19:48:30.288251 1767 raft_consensus.cc:359] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.288486 1767 raft_consensus.cc:385] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:30.288554 1767 raft_consensus.cc:740] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:30.288736 1767 consensus_queue.cc:260] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.290104 1767 ts_tablet_manager.cc:1434] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20251024 19:48:30.290468 1767 tablet_bootstrap.cc:492] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:30.291800 1766 tablet_bootstrap.cc:492] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:30.293656 1767 tablet_bootstrap.cc:654] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:30.290489 1586 leader_election.cc:343] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:30.297341 1586 leader_election.cc:343] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:30.299028 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b153f75caebb4d869e77529c9c9bb50d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:30.299264 1504 raft_consensus.cc:3060] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:30.303380 1504 raft_consensus.cc:2468] T b153f75caebb4d869e77529c9c9bb50d P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:30.329351 1768 tablet_bootstrap.cc:492] T b22b58404b8b441ba97ef536885a66e0 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:30.329515 1768 ts_tablet_manager.cc:1403] T b22b58404b8b441ba97ef536885a66e0 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.507s user 0.011s sys 0.022s
I20251024 19:48:30.332301 1768 raft_consensus.cc:359] T b22b58404b8b441ba97ef536885a66e0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.332499 1768 raft_consensus.cc:385] T b22b58404b8b441ba97ef536885a66e0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:30.332551 1768 raft_consensus.cc:740] T b22b58404b8b441ba97ef536885a66e0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:30.332774 1768 consensus_queue.cc:260] T b22b58404b8b441ba97ef536885a66e0 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.333205 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b153f75caebb4d869e77529c9c9bb50d" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:30.337780 1768 ts_tablet_manager.cc:1434] T b22b58404b8b441ba97ef536885a66e0 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251024 19:48:30.338063 1768 tablet_bootstrap.cc:492] T 92d78642772d44179e67d75c51b9da4e P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:30.357977 3760 consensus_queue.cc:1048] T ebcc09d8e029451d8fb6b6601ad16087 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:30.361486 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15de0d8aa9e545c09233186bbe361b97" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:30.361881 1507 raft_consensus.cc:3060] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:30.362705 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15de0d8aa9e545c09233186bbe361b97" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:30.364702 1768 tablet_bootstrap.cc:654] T 92d78642772d44179e67d75c51b9da4e P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:30.365437 1507 raft_consensus.cc:2468] T 15de0d8aa9e545c09233186bbe361b97 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:30.367072 3790 log.cc:826] T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233: Log is configured to *not* fsync() on all Append() calls
W20251024 19:48:30.365797 1369 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.003s sys 0.015s
W20251024 19:48:30.369122 1369 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s user 0.003s sys 0.015s
I20251024 19:48:30.369526 1774 raft_consensus.cc:2243] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:30.382323 1369 raft_consensus.cc:3060] T b22b58404b8b441ba97ef536885a66e0 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:30.386390 1774 pending_rounds.cc:70] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532: Trying to abort 1 pending ops.
I20251024 19:48:30.386502 1774 pending_rounds.cc:77] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214429425072091136 op_type: NO_OP noop_request { }
I20251024 19:48:30.386624 1774 raft_consensus.cc:2889] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251024 19:48:30.386747 1774 raft_consensus.cc:2272] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:30.389043 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_b9fe85eed58142c1a663db71a1f34756" } new_table_name: "default.Table_63dd4d4d3d1e486fa5d2a0b975446fde" modify_external_catalogs: true
I20251024 19:48:30.391882 1774 ts_tablet_manager.cc:1916] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 19:48:30.400548 1183 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9767f3fd8f3e48f6aa05bc054c1ddf67 consensus state has no leader
I20251024 19:48:30.402570 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1302) took 2331 ms. Trace:
I20251024 19:48:30.403106 1183 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1c68eae2ab444563993e0a40d23ce326" } modify_external_catalogs: true
I20251024 19:48:30.403343 1183 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_1c68eae2ab444563993e0a40d23ce326" } modify_external_catalogs: true
I20251024 19:48:30.404335 1776 rpcz_store.cc:276] 1024 19:48:28.071319 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:28.071382 (+ 63us) service_pool.cc:225] Handling call
1024 19:48:30.381783 (+2310401us) ts_tablet_manager.cc:1151] Deleting tablet 121cfcb3a6dc4ed18484ac8eeabc13f7
1024 19:48:30.385061 (+ 3278us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.402540 (+ 17479us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2310195}
I20251024 19:48:30.404960 1184 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_434a6ef23bea4f3991418b6ed2d9f635" } modify_external_catalogs: true
I20251024 19:48:30.405102 1184 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_434a6ef23bea4f3991418b6ed2d9f635" } modify_external_catalogs: true
W20251024 19:48:30.408911 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 121cfcb3a6dc4ed18484ac8eeabc13f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 121cfcb3a6dc4ed18484ac8eeabc13f7 already in progress: creating tablet
I20251024 19:48:30.411274 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1303) took 2339 ms. Trace:
I20251024 19:48:30.411645 1192 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_286aa7130af04dcfb71514e1f9add870" } new_table_name: "default.Table_1376faa2b6c64874b648e27d7311f6af" modify_external_catalogs: true
W20251024 19:48:30.419703 1192 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet daf07ab1ef05424b94c0611276979639 consensus state has no leader
I20251024 19:48:30.425977 1774 ts_tablet_manager.cc:1929] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:30.426254 1774 log.cc:1199] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/4a69c0918d2e4eed9d91b1913ec2c21e
I20251024 19:48:30.426718 1774 ts_tablet_manager.cc:1950] T 4a69c0918d2e4eed9d91b1913ec2c21e P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:30.427642 1329 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:54172 (request call id 186) took 1152 ms. Trace:
I20251024 19:48:30.428339 1188 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_28cc3637c8e04cd291b968fe9631ae13" } new_table_name: "default.Table_5fe0d79a415c4dc7b9708ed2203723a0" modify_external_catalogs: true
I20251024 19:48:30.427814 1329 rpcz_store.cc:276] 1024 19:48:29.275584 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:29.277636 (+ 2052us) service_pool.cc:225] Handling call
1024 19:48:30.370192 (+1092556us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x616000290254. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:30.427599 (+ 57407us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1264276352}
I20251024 19:48:30.433169 1776 rpcz_store.cc:276] 1024 19:48:28.071561 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:28.071619 (+ 58us) service_pool.cc:225] Handling call
1024 19:48:30.411131 (+2339512us) ts_tablet_manager.cc:1151] Deleting tablet 350d9403cc9242839d7dca5ff60ae6b4
1024 19:48:30.411143 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.411253 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2339342}
I20251024 19:48:30.436450 1187 master_service.cc:954] ReplaceTablet: received request to replace tablet 54bc7c7764584e11948d2831a4e5e0b6 from {username='slave'} at 127.0.0.1:51416
W20251024 19:48:30.437136 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 350d9403cc9242839d7dca5ff60ae6b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 350d9403cc9242839d7dca5ff60ae6b4 already in progress: creating tablet
I20251024 19:48:30.442539 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1305) took 2368 ms. Trace:
I20251024 19:48:30.442775 1776 rpcz_store.cc:276] 1024 19:48:28.073555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:28.073644 (+ 89us) service_pool.cc:225] Handling call
1024 19:48:30.441811 (+2368167us) ts_tablet_manager.cc:1151] Deleting tablet 404c38f8db2f43bfbc63e9ba2cee902c
1024 19:48:30.441823 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.442488 (+ 665us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367982}
W20251024 19:48:30.443660 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 404c38f8db2f43bfbc63e9ba2cee902c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 404c38f8db2f43bfbc63e9ba2cee902c already in progress: creating tablet
I20251024 19:48:30.451532 1190 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ce86da7ef95441c88843d4f307fdb809" } modify_external_catalogs: true
I20251024 19:48:30.451789 1190 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_ce86da7ef95441c88843d4f307fdb809" } modify_external_catalogs: true
I20251024 19:48:30.453039 1187 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_40abb33fa043417bb208476294761da9" } modify_external_catalogs: true
I20251024 19:48:30.453181 1187 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_40abb33fa043417bb208476294761da9" } modify_external_catalogs: true
I20251024 19:48:30.454705 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_9d038c302e9243bda195082a2e64fa57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:30.455263 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d038c302e9243bda195082a2e64fa57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:30.457000 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4c40dcd67a5b40dcae3436f75dd4aebc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:30.457486 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c40dcd67a5b40dcae3436f75dd4aebc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251024 19:48:30.458634 1586 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.252.252:36627, user_credentials={real_user=slave}} blocked reactor thread for 99498.9us
I20251024 19:48:30.458830 1189 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_29b0c38a3f5340b2877ffd4a9aa8312a" } modify_external_catalogs: true
W20251024 19:48:30.458947 1586 leader_election.cc:343] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:30.459033 1189 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_29b0c38a3f5340b2877ffd4a9aa8312a" } modify_external_catalogs: true
W20251024 19:48:30.459141 1586 consensus_peers.cc:597] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251024 19:48:30.459271 1586 leader_election.cc:343] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:30.459865 1186 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f53c5f387bd842608d81aca6d62631af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:30.460373 1186 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f53c5f387bd842608d81aca6d62631af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:30.461800 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f7a55abe972845af84f407a9c58a4540"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:30.462350 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7a55abe972845af84f407a9c58a4540 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:30.463457 3825 raft_consensus.cc:493] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:30.463593 3825 raft_consensus.cc:515] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:30.464186 3825 leader_election.cc:290] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:30.469237 1188 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb6b253a7c73485391366147a9bbc7f9 consensus state has no leader
I20251024 19:48:30.472096 1767 tablet_bootstrap.cc:492] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:30.472275 1767 ts_tablet_manager.cc:1403] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.182s user 0.007s sys 0.015s
I20251024 19:48:30.472445 1188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_4e591def703849b88d2a326b96e3d3c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:30.472988 1188 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e591def703849b88d2a326b96e3d3c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:30.473479 1767 raft_consensus.cc:359] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.473752 1767 raft_consensus.cc:385] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:30.473820 1767 raft_consensus.cc:740] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:30.473995 1767 consensus_queue.cc:260] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.475562 1767 ts_tablet_manager.cc:1434] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251024 19:48:30.475915 1767 tablet_bootstrap.cc:492] T d731403fbe0b4f92ac5899e13ea7bb19 P af3952de050140d383f6b0de7666fa97: Bootstrap starting.
I20251024 19:48:30.482708 1187 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet ee5356b5503c4425bf6648ade5f6330e
I20251024 19:48:30.493835 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:54172 (request call id 185) took 1218 ms. Trace:
I20251024 19:48:30.494119 1332 rpcz_store.cc:276] 1024 19:48:29.275302 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:29.275362 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:29.285807 (+ 10445us) spinlock_profiling.cc:243] Waited 1.36 ms on lock 0x61800011b48c. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f989102544e 00007f9890e20ac1 00007f9893944478 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8
1024 19:48:30.370160 (+1084353us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x616000290254. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:30.493803 (+123643us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1193095168}
I20251024 19:48:30.499210 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1304) took 2425 ms. Trace:
I20251024 19:48:30.499413 1776 rpcz_store.cc:276] 1024 19:48:28.073388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:28.073756 (+ 368us) service_pool.cc:225] Handling call
1024 19:48:30.498915 (+2425159us) ts_tablet_manager.cc:1151] Deleting tablet 19acf59898c8436e9d9fef98a7f07532
1024 19:48:30.498930 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.499180 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2425032}
I20251024 19:48:30.499984 1586 leader_election.cc:304] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
W20251024 19:48:30.500052 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 19acf59898c8436e9d9fef98a7f07532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19acf59898c8436e9d9fef98a7f07532 already in progress: creating tablet
I20251024 19:48:30.510710 1187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_448ada85f2f8471db370221787801578"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 19:48:30.511004 1613 tablet_service.cc:1552] Processing DeleteTablet for tablet ee5356b5503c4425bf6648ade5f6330e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:30.521198 3573 raft_consensus.cc:2804] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:30.521358 3573 raft_consensus.cc:697] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:30.521653 3573 consensus_queue.cc:237] T b153f75caebb4d869e77529c9c9bb50d P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:30.521836 1767 tablet_bootstrap.cc:654] T d731403fbe0b4f92ac5899e13ea7bb19 P af3952de050140d383f6b0de7666fa97: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:30.530122 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1306) took 2452 ms. Trace:
I20251024 19:48:30.530469 1776 rpcz_store.cc:276] 1024 19:48:28.077593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:28.077801 (+ 208us) service_pool.cc:225] Handling call
1024 19:48:30.529890 (+2452089us) ts_tablet_manager.cc:1151] Deleting tablet 6e419003a8644b0dbd4ce325a0b49d1e
1024 19:48:30.529900 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.530096 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2451719}
W20251024 19:48:30.531374 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 6e419003a8644b0dbd4ce325a0b49d1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e419003a8644b0dbd4ce325a0b49d1e already in progress: creating tablet
I20251024 19:48:30.537835 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1307) took 2439 ms. Trace:
I20251024 19:48:30.538789 1768 tablet_bootstrap.cc:492] T 92d78642772d44179e67d75c51b9da4e P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:30.538950 1768 ts_tablet_manager.cc:1403] T 92d78642772d44179e67d75c51b9da4e P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.201s user 0.002s sys 0.007s
I20251024 19:48:30.539824 1337 tablet_service.cc:1505] Processing CreateTablet for tablet 551479125efa4d58afb788651fa19c48 (DEFAULT_TABLE table=default.Table_5f5eee56374d49cd8a80b28f26f42bac [id=a3404453f85340068b15206987ffdc0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.539985 1768 raft_consensus.cc:359] T 92d78642772d44179e67d75c51b9da4e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.540170 1768 raft_consensus.cc:385] T 92d78642772d44179e67d75c51b9da4e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:30.540231 1768 raft_consensus.cc:740] T 92d78642772d44179e67d75c51b9da4e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:30.540591 1768 consensus_queue.cc:260] T 92d78642772d44179e67d75c51b9da4e P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.541253 1768 ts_tablet_manager.cc:1434] T 92d78642772d44179e67d75c51b9da4e P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251024 19:48:30.541639 1768 tablet_bootstrap.cc:492] T 0797d14e9fd846788164747523df0e2c P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:30.542555 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_5246796e5d1f4fc9b7e766d6bfa2ae29" } new_table_name: "default.Table_ea055d208a234cb8a8251660cf101752" modify_external_catalogs: true
I20251024 19:48:30.457067 3679 raft_consensus.cc:493] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:30.544235 3679 raft_consensus.cc:515] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:30.545778 3679 leader_election.cc:290] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:30.545902 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f88390f2534e4799a6eaa56aa4245e97" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:30.546393 1447 leader_election.cc:343] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:30.458300 1766 tablet_bootstrap.cc:654] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:30.559140 1766 tablet_bootstrap.cc:492] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:30.559424 1766 ts_tablet_manager.cc:1403] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.268s user 0.001s sys 0.006s
I20251024 19:48:30.560300 1766 raft_consensus.cc:359] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:30.560559 1766 raft_consensus.cc:385] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:30.560674 1766 raft_consensus.cc:740] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:30.560959 1775 ts_tablet_manager.cc:1929] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:30.560992 1766 consensus_queue.cc:260] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:30.561561 1766 ts_tablet_manager.cc:1434] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251024 19:48:30.561950 1766 tablet_bootstrap.cc:492] T 593f5600e4274fce97e7bd33f025569f P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:30.538450 1485 tablet_service.cc:1505] Processing CreateTablet for tablet f03048deb8b94b80924375bf3db068e9 (DEFAULT_TABLE table=default.Table_1175a8d8c6144a8d9479b68f16c35b05 [id=2131ad6aa3f040228db72a3cbe0e8bf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.563236 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f03048deb8b94b80924375bf3db068e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.534686 3760 consensus_queue.cc:1048] T b22b58404b8b441ba97ef536885a66e0 P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:30.569645 1337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 551479125efa4d58afb788651fa19c48. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:30.579638 1187 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_448ada85f2f8471db370221787801578 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:30.580648 1768 tablet_bootstrap.cc:654] T 0797d14e9fd846788164747523df0e2c P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:30.561393 1775 log.cc:1199] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/4100abc9ff12471e9a4959b6b24f7e99
I20251024 19:48:30.593861 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 551479125efa4d58afb788651fa19c48 (DEFAULT_TABLE table=default.Table_5f5eee56374d49cd8a80b28f26f42bac [id=a3404453f85340068b15206987ffdc0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.594345 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 551479125efa4d58afb788651fa19c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.594249 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 4f1d4542e7df4473a19f75f6037398b9 (DEFAULT_TABLE table=default.Table_5035151c0890403b9dc59e3973c94e76 [id=69f8936a0e804a989dda3c35852f7666]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.595347 1480 tablet_service.cc:1505] Processing CreateTablet for tablet 4f1d4542e7df4473a19f75f6037398b9 (DEFAULT_TABLE table=default.Table_5035151c0890403b9dc59e3973c94e76 [id=69f8936a0e804a989dda3c35852f7666]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.595274 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1d4542e7df4473a19f75f6037398b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.596038 1335 tablet_service.cc:1505] Processing CreateTablet for tablet daf07ab1ef05424b94c0611276979639 (DEFAULT_TABLE table=default.Table_286aa7130af04dcfb71514e1f9add870 [id=8a6be145b6d9418b8b081c173326f01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.596638 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf07ab1ef05424b94c0611276979639. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.604950 1473 tablet_service.cc:1505] Processing CreateTablet for tablet daf07ab1ef05424b94c0611276979639 (DEFAULT_TABLE table=default.Table_286aa7130af04dcfb71514e1f9add870 [id=8a6be145b6d9418b8b081c173326f01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.606031 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 8efbe6b9e59b4e91916bb6a3ea6f4c60 (DEFAULT_TABLE table=default.Table_d3cbf85ffa894f149214169d2e8c9977 [id=f1bfde166821406eab8d91dc94bad2ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.606480 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8efbe6b9e59b4e91916bb6a3ea6f4c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.606923 1768 tablet_bootstrap.cc:492] T 0797d14e9fd846788164747523df0e2c P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:30.608537 1768 ts_tablet_manager.cc:1403] T 0797d14e9fd846788164747523df0e2c P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.002s
I20251024 19:48:30.611402 1645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f88390f2534e4799a6eaa56aa4245e97" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
I20251024 19:48:30.613404 1768 raft_consensus.cc:359] T 0797d14e9fd846788164747523df0e2c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.613977 1185 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 92d78642772d44179e67d75c51b9da4e (table default.Table_52aa7a7ba04d4184bc9c976c07387929 [id=37332f920f4d4827b16f1c1c7bffda6d]). Expected version 2 got 0
I20251024 19:48:30.614785 1185 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_dbbb58e0778641c093acf3e7c24665bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:30.615320 1185 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbbb58e0778641c093acf3e7c24665bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:30.615746 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1d4542e7df4473a19f75f6037398b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.615968 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf07ab1ef05424b94c0611276979639. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.616257 1768 raft_consensus.cc:385] T 0797d14e9fd846788164747523df0e2c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:30.616328 1768 raft_consensus.cc:740] T 0797d14e9fd846788164747523df0e2c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:30.617013 1768 consensus_queue.cc:260] T 0797d14e9fd846788164747523df0e2c P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.619021 1768 ts_tablet_manager.cc:1434] T 0797d14e9fd846788164747523df0e2c P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251024 19:48:30.619415 1768 tablet_bootstrap.cc:492] T bc31e6c25ad14947af20abd766f12e78 P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:30.625082 1768 tablet_bootstrap.cc:654] T bc31e6c25ad14947af20abd766f12e78 P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
W20251024 19:48:30.646605 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 123305us
I20251024 19:48:30.647015 1586 leader_election.cc:304] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters: 43bec2f03fc849099e254f6335e9d532
I20251024 19:48:30.647730 3661 raft_consensus.cc:2804] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:30.647845 3661 raft_consensus.cc:697] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:30.648092 3661 consensus_queue.cc:237] T 15de0d8aa9e545c09233186bbe361b97 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.648809 3825 raft_consensus.cc:493] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:30.648923 3825 raft_consensus.cc:515] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.539098 1776 rpcz_store.cc:276] 1024 19:48:28.098533 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:28.099761 (+ 1228us) service_pool.cc:225] Handling call
1024 19:48:30.536635 (+2436874us) ts_tablet_manager.cc:1151] Deleting tablet 20e56d70049542f881f9490e4205dce3
1024 19:48:30.536775 (+ 140us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.537624 (+ 849us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2430578}
W20251024 19:48:30.649806 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet 20e56d70049542f881f9490e4205dce3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20e56d70049542f881f9490e4205dce3 already in progress: creating tablet
I20251024 19:48:30.650300 1768 tablet_bootstrap.cc:492] T bc31e6c25ad14947af20abd766f12e78 P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:30.650807 1768 ts_tablet_manager.cc:1403] T bc31e6c25ad14947af20abd766f12e78 P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.001s
I20251024 19:48:30.651769 1768 raft_consensus.cc:359] T bc31e6c25ad14947af20abd766f12e78 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.651924 1768 raft_consensus.cc:385] T bc31e6c25ad14947af20abd766f12e78 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:30.651978 1768 raft_consensus.cc:740] T bc31e6c25ad14947af20abd766f12e78 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:30.652042 1364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da1601a87524ef1a8b5069ba6823843" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:30.652159 1768 consensus_queue.cc:260] T bc31e6c25ad14947af20abd766f12e78 P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.652719 1768 ts_tablet_manager.cc:1434] T bc31e6c25ad14947af20abd766f12e78 P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251024 19:48:30.653097 1768 tablet_bootstrap.cc:492] T 8d1b6eaa353c4284a2b114b42437272c P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:30.653097 1366 raft_consensus.cc:3060] T bc31e6c25ad14947af20abd766f12e78 P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:30.656711 1768 tablet_bootstrap.cc:654] T 8d1b6eaa353c4284a2b114b42437272c P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:30.666477 1776 tablet_replica.cc:333] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:30.670935 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da1601a87524ef1a8b5069ba6823843" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:30.673349 1504 raft_consensus.cc:2468] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
W20251024 19:48:30.674621 1444 leader_election.cc:343] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:30.674772 1444 leader_election.cc:304] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:30.675758 3679 raft_consensus.cc:2749] T f88390f2534e4799a6eaa56aa4245e97 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:30.676707 1775 ts_tablet_manager.cc:1950] T 4100abc9ff12471e9a4959b6b24f7e99 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:30.677809 3845 consensus_queue.cc:1048] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:30.683473 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 8efbe6b9e59b4e91916bb6a3ea6f4c60 (DEFAULT_TABLE table=default.Table_d3cbf85ffa894f149214169d2e8c9977 [id=f1bfde166821406eab8d91dc94bad2ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.683954 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8efbe6b9e59b4e91916bb6a3ea6f4c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.693375 1338 tablet_service.cc:1505] Processing CreateTablet for tablet a54c6469dcea4b6da6d23c8395482c6f (DEFAULT_TABLE table=default.Table_34787bfb5ad443ff9c6ef9a773ddb6ff [id=d862b8cb002f407fab74b378d2501285]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.695678 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1217) took 5248 ms. Trace:
I20251024 19:48:30.697157 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a54c6469dcea4b6da6d23c8395482c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.712218 1339 tablet_service.cc:1505] Processing CreateTablet for tablet aea6f26c2b5144e78f3332ef67b14bb9 (DEFAULT_TABLE table=default.Table_09fd5426c34f40bc9ae783f13695e8bc [id=37625ba1414f4a1ebafa0561bd2e2058]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.712777 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aea6f26c2b5144e78f3332ef67b14bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.697400 1479 tablet_service.cc:1505] Processing CreateTablet for tablet a54c6469dcea4b6da6d23c8395482c6f (DEFAULT_TABLE table=default.Table_34787bfb5ad443ff9c6ef9a773ddb6ff [id=d862b8cb002f407fab74b378d2501285]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.709182 1766 tablet_bootstrap.cc:654] T 593f5600e4274fce97e7bd33f025569f P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:30.724433 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a54c6469dcea4b6da6d23c8395482c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.725832 1775 rpcz_store.cc:276] 1024 19:48:25.446889 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.446940 (+ 51us) service_pool.cc:225] Handling call
1024 19:48:30.242618 (+4795678us) ts_tablet_manager.cc:1151] Deleting tablet 4100abc9ff12471e9a4959b6b24f7e99
1024 19:48:30.242628 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.516097 (+273469us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:30.560834 (+ 44737us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:30.695641 (+134807us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4782281}
I20251024 19:48:30.726570 3825 leader_election.cc:290] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:30.729415 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 61b4135223a747198fcb4a6f052f4f08 (DEFAULT_TABLE table=default.Table_2d40dad65a34427ca08b38d576c386aa [id=545f5ceff0b44feea46b75496958d657]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.730312 3826 raft_consensus.cc:493] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:30.730432 3826 raft_consensus.cc:515] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.730532 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b4135223a747198fcb4a6f052f4f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.731050 3826 leader_election.cc:290] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:30.727752 1485 tablet_service.cc:1505] Processing CreateTablet for tablet aea6f26c2b5144e78f3332ef67b14bb9 (DEFAULT_TABLE table=default.Table_09fd5426c34f40bc9ae783f13695e8bc [id=37625ba1414f4a1ebafa0561bd2e2058]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.734179 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aea6f26c2b5144e78f3332ef67b14bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.735788 1489 tablet_service.cc:1505] Processing CreateTablet for tablet 61b4135223a747198fcb4a6f052f4f08 (DEFAULT_TABLE table=default.Table_2d40dad65a34427ca08b38d576c386aa [id=545f5ceff0b44feea46b75496958d657]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.736119 1489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b4135223a747198fcb4a6f052f4f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.737130 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 47c9fcf87a194c08a4e1de84dfc96f42 (DEFAULT_TABLE table=default.Table_a67503cccddd4a6e9d8ea3b30488cc14 [id=3b5b8389d9bf4203a0d2844ebac40304]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.737545 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c9fcf87a194c08a4e1de84dfc96f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.746665 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1220) took 5293 ms. Trace:
I20251024 19:48:30.749717 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 47c9fcf87a194c08a4e1de84dfc96f42 (DEFAULT_TABLE table=default.Table_a67503cccddd4a6e9d8ea3b30488cc14 [id=3b5b8389d9bf4203a0d2844ebac40304]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.750178 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c9fcf87a194c08a4e1de84dfc96f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.750399 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 20b6f6453f4d438a96e30fb30b20a8bd (DEFAULT_TABLE table=default.Table_42d61eaa29c64c7281f5cb3cf361b9e1 [id=d8d60aac53334cb3a71b27a010001481]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.750854 1475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20b6f6453f4d438a96e30fb30b20a8bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.751669 1350 tablet_service.cc:1505] Processing CreateTablet for tablet d1ac612a1a80485ea3ebb8e6e4e8e26e (DEFAULT_TABLE table=default.Table_312699096d58411cbc14672a7c9c7d9f [id=68f1bca8d63840529238be11ba2bbfce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.752049 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1ac612a1a80485ea3ebb8e6e4e8e26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.746981 1775 rpcz_store.cc:276] 1024 19:48:25.453275 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.453344 (+ 69us) service_pool.cc:225] Handling call
1024 19:48:30.734117 (+5280773us) ts_tablet_manager.cc:1151] Deleting tablet 977b4070419948d4910e2b69e01ff132
1024 19:48:30.734124 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.746636 (+ 12512us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5272227}
I20251024 19:48:30.773495 1346 tablet_service.cc:1505] Processing CreateTablet for tablet 20b6f6453f4d438a96e30fb30b20a8bd (DEFAULT_TABLE table=default.Table_42d61eaa29c64c7281f5cb3cf361b9e1 [id=d8d60aac53334cb3a71b27a010001481]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.783011 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20b6f6453f4d438a96e30fb30b20a8bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.784476 1485 tablet_service.cc:1505] Processing CreateTablet for tablet d1ac612a1a80485ea3ebb8e6e4e8e26e (DEFAULT_TABLE table=default.Table_312699096d58411cbc14672a7c9c7d9f [id=68f1bca8d63840529238be11ba2bbfce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.785010 1350 tablet_service.cc:1505] Processing CreateTablet for tablet e7a14fb9816f4120bc31caab65bd8273 (DEFAULT_TABLE table=default.Table_db1ad1d0cb0f4bfd9cb4fc16270c74d3 [id=447b25ec36984cbfa4c5200129daddd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.785616 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7a14fb9816f4120bc31caab65bd8273. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.785991 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1ac612a1a80485ea3ebb8e6e4e8e26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.827999 3679 raft_consensus.cc:493] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:30.828153 3679 raft_consensus.cc:515] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:30.828758 3679 leader_election.cc:290] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:30.832501 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0e5f7c28f0c47a88ca9872db2184ad0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
W20251024 19:48:30.833818 1191 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e111980a15894b5084d9384297389b5d consensus state has no leader
I20251024 19:48:30.834070 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0e5f7c28f0c47a88ca9872db2184ad0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:30.834439 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3da81c18d9264f38966944ad13f256d8" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:30.834637 1645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3da81c18d9264f38966944ad13f256d8" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:30.834998 1447 leader_election.cc:343] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:30.835314 1186 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_319540d9d2544798ac89ac3c8406f4ce" } new_table_name: "default.Table_629567165f0d4e95aeb4ac932453296c" modify_external_catalogs: true
W20251024 19:48:30.835893 1444 leader_election.cc:343] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:30.835999 1444 leader_election.cc:304] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10a9536207e343f188e0ef709081d04e; no voters: 43bec2f03fc849099e254f6335e9d532, af3952de050140d383f6b0de7666fa97
I20251024 19:48:30.836233 3760 raft_consensus.cc:2749] T 3da81c18d9264f38966944ad13f256d8 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251024 19:48:30.836633 1767 tablet_bootstrap.cc:492] T d731403fbe0b4f92ac5899e13ea7bb19 P af3952de050140d383f6b0de7666fa97: No bootstrap required, opened a new log
I20251024 19:48:30.836761 1767 ts_tablet_manager.cc:1403] T d731403fbe0b4f92ac5899e13ea7bb19 P af3952de050140d383f6b0de7666fa97: Time spent bootstrapping tablet: real 0.361s user 0.007s sys 0.006s
I20251024 19:48:30.839963 1192 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_875b4acd8c4f4e439fc174c155c4d248"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:30.840533 1192 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_875b4acd8c4f4e439fc174c155c4d248 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:30.842049 1369 raft_consensus.cc:3060] T 92d78642772d44179e67d75c51b9da4e P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:30.845722 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9684072368f14cc2942b4e240f90b0f2" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:30.846834 1507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9684072368f14cc2942b4e240f90b0f2" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:30.847057 1507 raft_consensus.cc:2468] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:30.847214 1191 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 2b2992a738b440d798f5e61cc45ec088 (table default.Table_8523c29419f0445aa4da2ffdc6480f1a [id=40394401a8a84431bfb5e2b1db774832]). Expected version 4 got 0
I20251024 19:48:30.847415 1191 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 0797d14e9fd846788164747523df0e2c (table default.Table_1f866642ce4a4ee885f169c4815bd1fb [id=a07755e138c84fd18c0fe72e81ab758b]). Expected version 1 got 0
I20251024 19:48:30.847568 1191 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 92d78642772d44179e67d75c51b9da4e (table default.Table_52aa7a7ba04d4184bc9c976c07387929 [id=37332f920f4d4827b16f1c1c7bffda6d]). Expected version 2 got 0
I20251024 19:48:30.849031 1191 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_85c4f5a27f584ee093fc8a6e9bfe279a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:30.849686 1191 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85c4f5a27f584ee093fc8a6e9bfe279a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:30.852098 3760 consensus_queue.cc:1048] T 92d78642772d44179e67d75c51b9da4e P 10a9536207e343f188e0ef709081d04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:30.853161 1767 raft_consensus.cc:359] T d731403fbe0b4f92ac5899e13ea7bb19 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.860918 1490 tablet_service.cc:1505] Processing CreateTablet for tablet f56f3d0e9e0a41ce91c0eeabe0b6dda1 (DEFAULT_TABLE table=default.Table_435492c6b1164482acacb1acbaa70afd [id=e2e000507ac74d958cd3c036cd90cc56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.861452 1776 raft_consensus.cc:2243] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:30.861825 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1218) took 5414 ms. Trace:
I20251024 19:48:30.862149 1776 raft_consensus.cc:2272] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:30.862090 1775 rpcz_store.cc:276] 1024 19:48:25.446975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.448387 (+ 1412us) service_pool.cc:225] Handling call
1024 19:48:30.779084 (+5330697us) ts_tablet_manager.cc:1151] Deleting tablet 5ace18d554f14acc892bc956c30e9979
1024 19:48:30.779098 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.861776 (+ 82678us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5316951}
I20251024 19:48:30.863832 1334 tablet_service.cc:1505] Processing CreateTablet for tablet f56f3d0e9e0a41ce91c0eeabe0b6dda1 (DEFAULT_TABLE table=default.Table_435492c6b1164482acacb1acbaa70afd [id=e2e000507ac74d958cd3c036cd90cc56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.865950 1490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f56f3d0e9e0a41ce91c0eeabe0b6dda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.866883 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1221) took 5384 ms. Trace:
I20251024 19:48:30.834231 1504 raft_consensus.cc:2468] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:30.859972 1485 tablet_service.cc:1505] Processing CreateTablet for tablet e7a14fb9816f4120bc31caab65bd8273 (DEFAULT_TABLE table=default.Table_db1ad1d0cb0f4bfd9cb4fc16270c74d3 [id=447b25ec36984cbfa4c5200129daddd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.869319 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7a14fb9816f4120bc31caab65bd8273. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.873729 1473 tablet_service.cc:1505] Processing CreateTablet for tablet 25645023adfc4adc838a27d78bc9a94c (DEFAULT_TABLE table=default.Table_21b929f19b444807ba23a7a062df8e34 [id=0b86fa21ecae403fa2fead4ba221dc0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.875797 1586 leader_election.cc:304] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:30.876078 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f56f3d0e9e0a41ce91c0eeabe0b6dda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.878669 1473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25645023adfc4adc838a27d78bc9a94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:30.876084 3858 consensus_queue.cc:1048] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:30.877046 1776 ts_tablet_manager.cc:1916] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:30.853543 1767 raft_consensus.cc:385] T d731403fbe0b4f92ac5899e13ea7bb19 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:30.888744 3825 raft_consensus.cc:2804] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:30.888841 3825 raft_consensus.cc:493] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:30.888899 3825 raft_consensus.cc:3060] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
W20251024 19:48:30.888981 1176 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet c70bfd01ae774245b42810722e508b46 (table default.Table_04e48889039f43688131634b37fc49e4 [id=3f87372b020a44a89a6c9b7687f19c98], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c70bfd01ae774245b42810722e508b46 consensus state has no leader
I20251024 19:48:30.891875 3825 raft_consensus.cc:515] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:30.892481 3825 leader_election.cc:290] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:30.892647 3825 consensus_queue.cc:1048] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:30.876523 1335 tablet_service.cc:1505] Processing CreateTablet for tablet 25645023adfc4adc838a27d78bc9a94c (DEFAULT_TABLE table=default.Table_21b929f19b444807ba23a7a062df8e34 [id=0b86fa21ecae403fa2fead4ba221dc0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:30.869283 1775 rpcz_store.cc:276] 1024 19:48:25.482313 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.497107 (+ 14794us) service_pool.cc:225] Handling call
1024 19:48:30.863272 (+5366165us) ts_tablet_manager.cc:1151] Deleting tablet bec3806757214c869cf1c4cecd7cc26e
1024 19:48:30.866755 (+ 3483us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.866863 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5344100}
I20251024 19:48:30.888087 1479 tablet_service.cc:1552] Processing DeleteTablet for tablet ee5356b5503c4425bf6648ade5f6330e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:30.900840 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1238) took 5069 ms. Trace:
I20251024 19:48:30.900954 1775 rpcz_store.cc:276] 1024 19:48:25.831055 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.831126 (+ 71us) service_pool.cc:225] Handling call
1024 19:48:30.900688 (+5069562us) ts_tablet_manager.cc:1151] Deleting tablet c084db163d874639b4385f41e9efaa18
1024 19:48:30.900702 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.900818 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5023416}
I20251024 19:48:30.901610 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1239) took 5070 ms. Trace:
I20251024 19:48:30.901707 1775 rpcz_store.cc:276] 1024 19:48:25.831162 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.831222 (+ 60us) service_pool.cc:225] Handling call
1024 19:48:30.901467 (+5070245us) ts_tablet_manager.cc:1151] Deleting tablet d72f4761820e430cb1a97d83cd30c75d
1024 19:48:30.901477 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.901591 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5019788}
I20251024 19:48:30.902410 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1240) took 5071 ms. Trace:
I20251024 19:48:30.902505 1775 rpcz_store.cc:276] 1024 19:48:25.831278 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.831330 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:30.902285 (+5070955us) ts_tablet_manager.cc:1151] Deleting tablet 3c608ac55e9e42bc9c9b27beec717e30
1024 19:48:30.902296 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.902380 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5020537}
I20251024 19:48:30.902992 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1241) took 5071 ms. Trace:
I20251024 19:48:30.903082 1775 rpcz_store.cc:276] 1024 19:48:25.831518 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.846723 (+ 15205us) service_pool.cc:225] Handling call
1024 19:48:30.902878 (+5056155us) ts_tablet_manager.cc:1151] Deleting tablet b383a83425654723bd1d9c3a39b187be
1024 19:48:30.902889 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.902974 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4959405}
I20251024 19:48:30.909641 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1242) took 5078 ms. Trace:
W20251024 19:48:30.918965 1186 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bfe5aa5bd9844560874613d445ba5cc1 consensus state has no leader
I20251024 19:48:30.886871 3860 tablet.cc:1722] T c512ec26e3874cd7869216b1fb90fa54 P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:30.888619 1342 tablet_service.cc:1552] Processing DeleteTablet for tablet ee5356b5503c4425bf6648ade5f6330e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:30.926054 1774 tablet_replica.cc:333] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:30.926241 1774 raft_consensus.cc:2243] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:30.926432 1774 raft_consensus.cc:2272] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:30.927940 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet e38ef2ac19d14f9c91fa8abe86cc7987 (table default.Table_6bb9ce85a6e341e7b2cc6d9ffc2c0127 [id=50a4f64cc7504ea190832dd265c81caf]) successfully deleted
W20251024 19:48:30.928207 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 28f93be6762146398421ac19d2e84e0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28f93be6762146398421ac19d2e84e0c already in progress: creating tablet
I20251024 19:48:30.929258 1179 catalog_manager.cc:4985] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): tablet 4100abc9ff12471e9a4959b6b24f7e99 (table default.Table_da24358ca9544a0e805bb937715888f9 [id=0e3c3f6ff360497c93fac944f351702c]) successfully deleted
I20251024 19:48:30.896025 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25645023adfc4adc838a27d78bc9a94c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:30.929970 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 977b4070419948d4910e2b69e01ff132 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 977b4070419948d4910e2b69e01ff132 already in progress: creating tablet
I20251024 19:48:30.930088 3825 consensus_queue.cc:1048] T 0797d14e9fd846788164747523df0e2c P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:30.930528 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 5ace18d554f14acc892bc956c30e9979 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ace18d554f14acc892bc956c30e9979 already in progress: creating tablet
W20251024 19:48:30.930859 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet bec3806757214c869cf1c4cecd7cc26e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bec3806757214c869cf1c4cecd7cc26e already in progress: creating tablet
W20251024 19:48:30.931047 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet c084db163d874639b4385f41e9efaa18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c084db163d874639b4385f41e9efaa18 already in progress: creating tablet
W20251024 19:48:30.931242 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet d72f4761820e430cb1a97d83cd30c75d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d72f4761820e430cb1a97d83cd30c75d already in progress: creating tablet
W20251024 19:48:30.931427 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet 3c608ac55e9e42bc9c9b27beec717e30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c608ac55e9e42bc9c9b27beec717e30 already in progress: creating tablet
W20251024 19:48:30.931594 1179 catalog_manager.cc:4965] TS 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045): delete failed for tablet b383a83425654723bd1d9c3a39b187be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b383a83425654723bd1d9c3a39b187be already in progress: creating tablet
I20251024 19:48:30.931804 1586 leader_election.cc:304] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:30.932116 3825 raft_consensus.cc:2804] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:30.932195 3825 raft_consensus.cc:493] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:30.932261 3825 raft_consensus.cc:3060] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:30.932695 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet 4a69c0918d2e4eed9d91b1913ec2c21e (table default.Table_8b8ee145388d4588b50306be38739650 [id=8f925cc7435543aa8aacba668bd66b52]) successfully deleted
W20251024 19:48:30.932888 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 300952047a324a25a7c687eff20d6e56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 300952047a324a25a7c687eff20d6e56 already in progress: creating tablet
I20251024 19:48:30.935611 3825 raft_consensus.cc:515] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.936348 3825 leader_election.cc:290] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:30.940758 1189 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 4ee9fb177b4a4dbc8fcd2c9f0b6ee699
I20251024 19:48:30.942023 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ee9fb177b4a4dbc8fcd2c9f0b6ee699 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:30.948799 1586 leader_election.cc:304] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:30.949287 3828 raft_consensus.cc:2804] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:30.949419 3828 raft_consensus.cc:493] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:30.949555 3828 raft_consensus.cc:3060] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:30.949900 1183 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 529580599bfa4800973b8540ab13d22a
W20251024 19:48:30.950690 1586 leader_election.cc:343] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:30.951176 1586 consensus_peers.cc:597] T 8d1b6eaa353c4284a2b114b42437272c P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:30.951927 3858 consensus_queue.cc:1048] T bc31e6c25ad14947af20abd766f12e78 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:30.953706 1188 catalog_manager.cc:5649] T 774a2de2e3764c54b72acfb952ed3e11 P af3952de050140d383f6b0de7666fa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to af3952de050140d383f6b0de7666fa97 (127.0.252.195). New cstate: current_term: 1 leader_uuid: "af3952de050140d383f6b0de7666fa97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } health_report { overall_health: UNKNOWN } } }
I20251024 19:48:30.954246 1188 catalog_manager.cc:5741] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119) does not have the latest schema for tablet 774a2de2e3764c54b72acfb952ed3e11 (table default.Table_5fb8e365cba24ded8061c38d9c695ff8 [id=1a8ca3395d7b471fa354154f2468080f]). Expected version 1 got 0
W20251024 19:48:30.961778 1586 consensus_peers.cc:597] T 6ef2add387da44868dbd2b5724ee7d86 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:30.962421 3858 consensus_queue.cc:1048] T 3e6f8764e9ae472b9ac35a032182aed1 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:30.965026 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet 529580599bfa4800973b8540ab13d22a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:54246
W20251024 19:48:30.964686 1586 leader_election.cc:343] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:30.966080 1586 leader_election.cc:343] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:30.966813 1586 consensus_peers.cc:597] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:30.969712 3573 raft_consensus.cc:493] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:30.969884 3573 raft_consensus.cc:515] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:30.970506 3573 leader_election.cc:290] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:30.974560 1184 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 799780bf376843499c659df4cd2b4aa1
I20251024 19:48:30.975261 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 786f0e890bb6410da9172eee32504be1
I20251024 19:48:30.977525 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet 799780bf376843499c659df4cd2b4aa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:30.977820 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 786f0e890bb6410da9172eee32504be1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:30.979094 1510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da1601a87524ef1a8b5069ba6823843" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:30.979277 1510 raft_consensus.cc:3060] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:30.985267 1508 raft_consensus.cc:1275] T 2671108031db449cae2135835fc5f739 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Refusing update from remote peer af3952de050140d383f6b0de7666fa97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 19:48:30.994370 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "373aa543e1284fbc945ca802d9eddf59" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e" is_pre_election: true
I20251024 19:48:30.994623 1508 raft_consensus.cc:2468] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 0.
I20251024 19:48:30.999075 1502 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9684072368f14cc2942b4e240f90b0f2" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:31.003321 1502 raft_consensus.cc:3060] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:31.008337 1188 catalog_manager.cc:6194] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: default.Table_e97cbcd597f2420d99b0c790803e4dff [id=a37609f4cf31400db7b95becebfb4222] alter complete (version 1)
I20251024 19:48:31.011008 1624 tablet_service.cc:1552] Processing DeleteTablet for tablet 7da1601a87524ef1a8b5069ba6823843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:05 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:31.011240 1625 tablet_service.cc:1552] Processing DeleteTablet for tablet 373aa543e1284fbc945ca802d9eddf59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:11 UTC) from {username='slave'} at 127.0.0.1:54246
I20251024 19:48:31.015834 1776 ts_tablet_manager.cc:1929] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:30.909983 1775 rpcz_store.cc:276] 1024 19:48:25.831593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:25.847374 (+ 15781us) service_pool.cc:225] Handling call
1024 19:48:30.908472 (+5061098us) ts_tablet_manager.cc:1151] Deleting tablet e0228d5b788e4c42bf4ecce876ecc3f4
1024 19:48:30.908521 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.909607 (+ 1086us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4964894}
I20251024 19:48:31.022603 1338 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e6f8764e9ae472b9ac35a032182aed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:31.015774 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 97300d4eb8214d2bad6db432db47e129 (DEFAULT_TABLE table=default.Table_4102ac21c0a345da986f9f8f4d66a1b4 [id=c7994531a5904b1180d7ecabc1b39b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.025926 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97300d4eb8214d2bad6db432db47e129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.025887 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 0bd17c969314425799a12e2cb878d957 (DEFAULT_TABLE table=default.Table_df274196c7264f7aa9c5e59b7b50d657 [id=6a0cdea1153c4a9d957aa87f52d8c974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.026149 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd17c969314425799a12e2cb878d957. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.032007 1510 raft_consensus.cc:2468] T 7da1601a87524ef1a8b5069ba6823843 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:31.032189 1776 log.cc:1199] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/4a69c0918d2e4eed9d91b1913ec2c21e
I20251024 19:48:31.029536 1502 raft_consensus.cc:2468] T 9684072368f14cc2942b4e240f90b0f2 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:31.032838 1776 ts_tablet_manager.cc:1950] T 4a69c0918d2e4eed9d91b1913ec2c21e P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:31.033730 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1312) took 2887 ms. Trace:
I20251024 19:48:31.034021 1776 rpcz_store.cc:276] 1024 19:48:28.146475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:28.146547 (+ 72us) service_pool.cc:225] Handling call
1024 19:48:30.665921 (+2519374us) ts_tablet_manager.cc:1151] Deleting tablet 4a69c0918d2e4eed9d91b1913ec2c21e
1024 19:48:30.666425 (+ 504us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:30.861927 (+195502us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x6160001a2054. stack: 00007fea3c4169f6 00007fea3c4167b0 00007fea3b3f23ad 0000000000983f04 0000000000983ecd 000000000098389a 00007fea482dc100 00007fea49629ae3 00007fea4c1e6287 00007fea4c2192d4 00007fea4c1f9d70 00007fea4c1f9afd 0000000000a8867c 00007fea3c45cd7f 00007fea3c46004b 00007fea3c45fec8
1024 19:48:30.971107 (+109180us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:31.015729 (+ 44622us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:31.033711 (+ 17982us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":447841024,"tablet-delete.queue_time_us":2519125}
I20251024 19:48:31.034564 1776 tablet_replica.cc:333] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:31.034598 1188 catalog_manager.cc:5741] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103) does not have the latest schema for tablet 92d78642772d44179e67d75c51b9da4e (table default.Table_52aa7a7ba04d4184bc9c976c07387929 [id=37332f920f4d4827b16f1c1c7bffda6d]). Expected version 2 got 0
I20251024 19:48:31.034808 1776 raft_consensus.cc:2243] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 19:48:31.034967 1776 raft_consensus.cc:2272] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:31.035389 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9684072368f14cc2942b4e240f90b0f2" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:31.035472 1776 ts_tablet_manager.cc:1916] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:31.036279 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet 4a69c0918d2e4eed9d91b1913ec2c21e (table default.Table_8b8ee145388d4588b50306be38739650 [id=8f925cc7435543aa8aacba668bd66b52]) successfully deleted
I20251024 19:48:31.038378 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 76a4c7ec8e514b26b6b270fdab8b9b5a (DEFAULT_TABLE table=default.Table_81498524d757401faaff766b9db57c27 [id=2613adda6a1c438bae1b3db87fb910d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.038810 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a4c7ec8e514b26b6b270fdab8b9b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.041793 1370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7da1601a87524ef1a8b5069ba6823843" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:31.042297 1774 ts_tablet_manager.cc:1916] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:31.045372 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 56e63d72f666493785eaa8ac46e71e91 (DEFAULT_TABLE table=default.Table_2fc89eee3d764c409f66027b6b878714 [id=4c8fa97560434affb57bc85e64dad190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.016628 3828 raft_consensus.cc:515] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:31.016772 3825 consensus_queue.cc:1048] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 19:48:31.045964 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56e63d72f666493785eaa8ac46e71e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.016644 3879 tablet.cc:1722] T 1999b751e9344dcf81d26226c0589c8f P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:31.017796 3661 consensus_queue.cc:1048] T 2671108031db449cae2135835fc5f739 P af3952de050140d383f6b0de7666fa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 19:48:31.047439 3871 log.cc:927] Time spent T 1999b751e9344dcf81d26226c0589c8f P 43bec2f03fc849099e254f6335e9d532: Append to log took a long time: real 0.160s user 0.000s sys 0.000s
I20251024 19:48:31.052117 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "373aa543e1284fbc945ca802d9eddf59" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532" is_pre_election: true
I20251024 19:48:31.053159 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 8d00c2d71fe7416c80a690f7c48e03d1 (DEFAULT_TABLE table=default.Table_cc2d8128e49f486bbc9cf83bb4b112cc [id=ef8ef07b3ff047579e125b267b7de277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.054674 1623 tablet_service.cc:1505] Processing CreateTablet for tablet 29f6d72182bb4f289e8f4b965da038b1 (DEFAULT_TABLE table=default.Table_0b75e1eb5e35490ab2947463e6788059 [id=4c5601b2d4c6479186929fd28ab240e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.055351 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d00c2d71fe7416c80a690f7c48e03d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.057778 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f6d72182bb4f289e8f4b965da038b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.060117 1611 tablet_service.cc:1505] Processing CreateTablet for tablet bd345b7c5f304004a908c99cb511e4d1 (DEFAULT_TABLE table=default.Table_353d0c63ddcc41d2aa242bb7e8f89f14 [id=a619e35185574efcb1f578d2e5610c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.060420 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd345b7c5f304004a908c99cb511e4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.061794 1612 tablet_service.cc:1505] Processing CreateTablet for tablet db4dd2ff754342359ffde52ec47c5e62 (DEFAULT_TABLE table=default.Table_2ca6f1a424034ecdb01cef12ae91756c [id=301745a621a043b883dab93342a76553]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.062100 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db4dd2ff754342359ffde52ec47c5e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.062880 1625 tablet_service.cc:1505] Processing CreateTablet for tablet f7534929046a485bbf461ddd0b468ebb (DEFAULT_TABLE table=default.Table_862cb2dda48a4dbd954b5d852fb0e5fb [id=d5b82997e3bb400c8820bcc2b8d58f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.063158 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7534929046a485bbf461ddd0b468ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.067699 1630 tablet_service.cc:1505] Processing CreateTablet for tablet cab80a02b1264bf1b4e40f459a467be3 (DEFAULT_TABLE table=default.Table_535b9a93bdda424a81bbdc6c08c3450f [id=9cb0b1b40eaf46f798e6f018727f8d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.068027 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab80a02b1264bf1b4e40f459a467be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.073073 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 79ad354b4f844055baaa997cee6a3d3f (DEFAULT_TABLE table=default.Table_7040850ac4da4483ad7cc6d41cc17eb4 [id=bd49e5c0642a4bd1b983f4c534887644]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.073513 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79ad354b4f844055baaa997cee6a3d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.074139 1614 tablet_service.cc:1505] Processing CreateTablet for tablet 80be570302354c2ba5be47f22ef5e083 (DEFAULT_TABLE table=default.Table_db05043847df4217a12a2524e5dc05ce [id=959edfd20d594810ba953638a3cdc7eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.074515 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80be570302354c2ba5be47f22ef5e083. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.076620 1622 tablet_service.cc:1505] Processing CreateTablet for tablet 175a2aa798ec40e6a57a4b63be7e180b (DEFAULT_TABLE table=default.Table_b1945a1bef6a40fb8dd92f10bcb2ff10 [id=52bce2dfe0c543faa2fbc2b0592e4bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.076960 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 175a2aa798ec40e6a57a4b63be7e180b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.082459 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 701c7094a6bc41bb87483d150e7d0727 (DEFAULT_TABLE table=default.Table_22c143aabac54b76afbf3cf41fac7d57 [id=cf1bf27deea242d392906a54987ef11b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.082844 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 701c7094a6bc41bb87483d150e7d0727. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.090482 1630 tablet_service.cc:1505] Processing CreateTablet for tablet a8f7c5db8eb84671be28c5368c345f37 (DEFAULT_TABLE table=default.Table_d65da5a2f7214fdca166986046157c60 [id=7a8e293425f641daae6afc0de8c01caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.090883 1630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8f7c5db8eb84671be28c5368c345f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.092428 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 40e6298cb7fa414f88a9d2cad553de60 (DEFAULT_TABLE table=default.Table_2ff196bb7bb24e7e9bdecaa140baf000 [id=b18743b13edb4f7e857d04385aa1a6ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.092852 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40e6298cb7fa414f88a9d2cad553de60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.018038 1586 leader_election.cc:304] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:31.020102 1624 tablet_service.cc:1505] Processing CreateTablet for tablet e4ae4b207f524de8a3629590bd9882db (DEFAULT_TABLE table=default.Table_f997c14171b3466f9c2c388c542735bd [id=dabcbcba9c0244ae86dc6fdae6325454]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.099248 3661 raft_consensus.cc:2804] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 19:48:31.099337 3661 raft_consensus.cc:493] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 19:48:31.099423 3661 raft_consensus.cc:3060] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:31.099452 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4ae4b207f524de8a3629590bd9882db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.022396 1613 tablet_service.cc:1505] Processing CreateTablet for tablet 7212e488655f4f5390eb683749f41994 (DEFAULT_TABLE table=default.Table_5bf3fd6ad2884dd8a4dcd772d793798d [id=e00cb3c037ed43d9afe9554146273f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.101861 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7212e488655f4f5390eb683749f41994. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.102380 3661 raft_consensus.cc:515] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:31.102988 3661 leader_election.cc:290] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:31.103168 1586 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.252.194:34045, user_credentials={real_user=slave}} blocked reactor thread for 85156.4us
I20251024 19:48:31.023671 1619 tablet_service.cc:1505] Processing CreateTablet for tablet c93eccd2a95b4a79a94d5e35f4414bb3 (DEFAULT_TABLE table=default.Table_5f7d0b8e722a457ab176bb08207e178c [id=2e682fd1a8d04539b3d22c8d7fde59d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.105824 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c93eccd2a95b4a79a94d5e35f4414bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.024595 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 12bcf196c6b4429c9cade601782d27ef (DEFAULT_TABLE table=default.Table_c5daa131860c4f369dae0280b46a2165 [id=8492e7b523464e07abd051ee41cf63bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.109915 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12bcf196c6b4429c9cade601782d27ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.110340 1586 leader_election.cc:304] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:31.110934 3573 raft_consensus.cc:2804] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:31.111032 3573 raft_consensus.cc:697] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:31.111249 3573 consensus_queue.cc:237] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:31.112478 1586 leader_election.cc:304] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a9536207e343f188e0ef709081d04e, af3952de050140d383f6b0de7666fa97; no voters:
I20251024 19:48:31.113224 3661 raft_consensus.cc:2804] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 19:48:31.113337 3661 raft_consensus.cc:697] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [term 1 LEADER]: Becoming Leader. State: Replica: af3952de050140d383f6b0de7666fa97, State: Running, Role: LEADER
I20251024 19:48:31.113598 3661 consensus_queue.cc:237] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:31.123327 3828 leader_election.cc:290] T d0e5f7c28f0c47a88ca9872db2184ad0 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Requested vote from peers 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
W20251024 19:48:31.126432 1586 leader_election.cc:343] T 9684072368f14cc2942b4e240f90b0f2 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:31.126890 1586 leader_election.cc:343] T 7da1601a87524ef1a8b5069ba6823843 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:31.127327 1586 consensus_peers.cc:597] T ee5356b5503c4425bf6648ade5f6330e P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:31.128080 1766 tablet_bootstrap.cc:492] T 593f5600e4274fce97e7bd33f025569f P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:31.128227 1766 ts_tablet_manager.cc:1403] T 593f5600e4274fce97e7bd33f025569f P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.566s user 0.009s sys 0.014s
W20251024 19:48:31.128211 1586 leader_election.cc:343] T 373aa543e1284fbc945ca802d9eddf59 P af3952de050140d383f6b0de7666fa97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Illegal state: must be running to vote when last-logged opid is not known
W20251024 19:48:31.128525 1586 consensus_peers.cc:597] T 7deb97be82964ea0b2fe9caa867a6460 P af3952de050140d383f6b0de7666fa97 -> Peer 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): Couldn't send request to peer 43bec2f03fc849099e254f6335e9d532. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251024 19:48:31.133898 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1244) took 4912 ms. Trace:
I20251024 19:48:31.135427 1508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "373aa543e1284fbc945ca802d9eddf59" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:31.135593 1508 raft_consensus.cc:3060] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:31.134116 1775 rpcz_store.cc:276] 1024 19:48:26.221895 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.221947 (+ 52us) service_pool.cc:225] Handling call
1024 19:48:31.020658 (+4798711us) ts_tablet_manager.cc:1151] Deleting tablet 5d62c5c336e14c65bf6734c66e9be198
1024 19:48:31.133794 (+113136us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:31.133882 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4798498}
I20251024 19:48:31.137279 1366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "373aa543e1284fbc945ca802d9eddf59" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
W20251024 19:48:31.142689 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 4ec69ffe2c024d1a8596a93183d1458b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ec69ffe2c024d1a8596a93183d1458b already in progress: creating tablet
W20251024 19:48:31.142966 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet e6fd158e8b0c4cc78cf7fd3b34d5f104 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6fd158e8b0c4cc78cf7fd3b34d5f104 already in progress: creating tablet
W20251024 19:48:31.143196 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 182314f0f3074150b139cb2476976e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 182314f0f3074150b139cb2476976e1a already in progress: creating tablet
W20251024 19:48:31.143399 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet a403201fab014ea98734c3e80649e088 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a403201fab014ea98734c3e80649e088 already in progress: creating tablet
W20251024 19:48:31.143638 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 2b119e8ceae04852964513ebfa7aa31f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b119e8ceae04852964513ebfa7aa31f already in progress: creating tablet
W20251024 19:48:31.143945 1179 catalog_manager.cc:4965] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): delete failed for tablet 398e73d6d576487e968c9083175965aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 398e73d6d576487e968c9083175965aa already in progress: creating tablet
I20251024 19:48:31.144094 1774 ts_tablet_manager.cc:1929] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:31.144234 1774 log.cc:1199] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-0/wal/wals/ee5356b5503c4425bf6648ade5f6330e
I20251024 19:48:31.137369 1372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0e5f7c28f0c47a88ca9872db2184ad0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bec2f03fc849099e254f6335e9d532"
I20251024 19:48:31.143157 1504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0e5f7c28f0c47a88ca9872db2184ad0" candidate_uuid: "af3952de050140d383f6b0de7666fa97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a9536207e343f188e0ef709081d04e"
I20251024 19:48:31.157627 1504 raft_consensus.cc:3060] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Advancing to term 1
I20251024 19:48:31.160351 1504 raft_consensus.cc:2468] T d0e5f7c28f0c47a88ca9872db2184ad0 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:31.143440 1766 raft_consensus.cc:359] T 593f5600e4274fce97e7bd33f025569f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:31.161011 1766 raft_consensus.cc:385] T 593f5600e4274fce97e7bd33f025569f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:31.161062 1766 raft_consensus.cc:740] T 593f5600e4274fce97e7bd33f025569f P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:31.161263 1766 consensus_queue.cc:260] T 593f5600e4274fce97e7bd33f025569f P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:31.161844 1766 ts_tablet_manager.cc:1434] T 593f5600e4274fce97e7bd33f025569f P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.034s user 0.001s sys 0.001s
I20251024 19:48:31.162148 1766 tablet_bootstrap.cc:492] T 74029a7d7cdc4503a833c1758887b684 P 10a9536207e343f188e0ef709081d04e: Bootstrap starting.
I20251024 19:48:31.156736 1508 raft_consensus.cc:2468] T 373aa543e1284fbc945ca802d9eddf59 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af3952de050140d383f6b0de7666fa97 in term 1.
I20251024 19:48:31.139542 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1243) took 4917 ms. Trace:
I20251024 19:48:31.164336 1775 rpcz_store.cc:276] 1024 19:48:26.221750 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.221946 (+ 196us) service_pool.cc:225] Handling call
1024 19:48:31.139393 (+4917447us) ts_tablet_manager.cc:1151] Deleting tablet bcdd47a64e2142cda1f2d20afebd4ce5
1024 19:48:31.139402 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:31.139525 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4916046}
I20251024 19:48:31.165060 1775 tablet_replica.cc:333] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e: stopping tablet replica
I20251024 19:48:31.169049 1775 raft_consensus.cc:2243] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [term 1 LEADER]: Raft consensus shutting down.
I20251024 19:48:31.169510 1775 raft_consensus.cc:2272] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 19:48:31.171432 1766 tablet_bootstrap.cc:654] T 74029a7d7cdc4503a833c1758887b684 P 10a9536207e343f188e0ef709081d04e: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:31.173311 1775 ts_tablet_manager.cc:1916] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 19:48:31.178036 1774 ts_tablet_manager.cc:1950] T ee5356b5503c4425bf6648ade5f6330e P 43bec2f03fc849099e254f6335e9d532: Deleting consensus metadata
I20251024 19:48:31.178871 1774 tablet_replica.cc:333] T 3e6f8764e9ae472b9ac35a032182aed1 P 43bec2f03fc849099e254f6335e9d532: stopping tablet replica
I20251024 19:48:31.180501 1776 ts_tablet_manager.cc:1929] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251024 19:48:31.180663 1776 log.cc:1199] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-2/wal/wals/ab488e4971874a16b2e052c84c250777
I20251024 19:48:31.192073 1767 raft_consensus.cc:740] T d731403fbe0b4f92ac5899e13ea7bb19 P af3952de050140d383f6b0de7666fa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3952de050140d383f6b0de7666fa97, State: Initialized, Role: FOLLOWER
I20251024 19:48:31.201347 1767 consensus_queue.cc:260] T d731403fbe0b4f92ac5899e13ea7bb19 P af3952de050140d383f6b0de7666fa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
W20251024 19:48:31.202669 1179 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.252.193:36103, user_credentials={real_user=slave}} blocked reactor thread for 57908.5us
I20251024 19:48:31.203656 1179 catalog_manager.cc:4985] TS 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103): tablet ee5356b5503c4425bf6648ade5f6330e (table default.Table_40abb33fa043417bb208476294761da9 [id=239fc223508c45a980b6254769bc8c12]) successfully deleted
I20251024 19:48:31.208117 3888 tablet.cc:1722] T 1999b751e9344dcf81d26226c0589c8f P 10a9536207e343f188e0ef709081d04e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:31.211730 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet ebcc09d8e029451d8fb6b6601ad16087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:04 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:31.213224 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet b22b58404b8b441ba97ef536885a66e0 with delete_type TABLET_DATA_DELETED (replaced by tablet f37a01dc3d5f458d81605f2c41fa27dd) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:31.213372 1775 ts_tablet_manager.cc:1929] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251024 19:48:31.213550 1775 log.cc:1199] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e: Deleting WAL directory at /tmp/dist-test-taskc9NInd/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761335271430102-1011-0/minicluster-data/ts-1/wal/wals/c5cbce9e4b61451ba7f458df922be759
I20251024 19:48:31.213989 1775 ts_tablet_manager.cc:1950] T c5cbce9e4b61451ba7f458df922be759 P 10a9536207e343f188e0ef709081d04e: Deleting consensus metadata
I20251024 19:48:31.214350 1775 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46462 (request call id 1245) took 4880 ms. Trace:
I20251024 19:48:31.214442 1775 rpcz_store.cc:276] 1024 19:48:26.333800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:26.375017 (+ 41217us) service_pool.cc:225] Handling call
1024 19:48:31.165009 (+4789992us) ts_tablet_manager.cc:1151] Deleting tablet c5cbce9e4b61451ba7f458df922be759
1024 19:48:31.165020 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:31.195135 (+ 30115us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:31.213258 (+ 18123us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:31.214330 (+ 1072us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4789745}
I20251024 19:48:31.217878 1766 tablet_bootstrap.cc:492] T 74029a7d7cdc4503a833c1758887b684 P 10a9536207e343f188e0ef709081d04e: No bootstrap required, opened a new log
I20251024 19:48:31.218039 1766 ts_tablet_manager.cc:1403] T 74029a7d7cdc4503a833c1758887b684 P 10a9536207e343f188e0ef709081d04e: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.001s
I20251024 19:48:31.219163 1766 raft_consensus.cc:359] T 74029a7d7cdc4503a833c1758887b684 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:31.219523 1766 raft_consensus.cc:385] T 74029a7d7cdc4503a833c1758887b684 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:31.219655 1766 raft_consensus.cc:740] T 74029a7d7cdc4503a833c1758887b684 P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a9536207e343f188e0ef709081d04e, State: Initialized, Role: FOLLOWER
I20251024 19:48:31.219995 1766 consensus_queue.cc:260] T 74029a7d7cdc4503a833c1758887b684 P 10a9536207e343f188e0ef709081d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } }
I20251024 19:48:31.220875 1766 ts_tablet_manager.cc:1434] T 74029a7d7cdc4503a833c1758887b684 P 10a9536207e343f188e0ef709081d04e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251024 19:48:31.227067 3790 log.cc:927] Time spent T 00000000000000000000000000000000 P 82e82847f8904b59b03f446b7fce6233: Append to log took a long time: real 0.105s user 0.004s sys 0.000s
I20251024 19:48:31.230340 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ee9fb177b4a4dbc8fcd2c9f0b6ee699 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:31.230644 1489 tablet_service.cc:1552] Processing DeleteTablet for tablet 529580599bfa4800973b8540ab13d22a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:31.233418 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 799780bf376843499c659df4cd2b4aa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:31.235102 1490 tablet_service.cc:1552] Processing DeleteTablet for tablet 786f0e890bb6410da9172eee32504be1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:46462
I20251024 19:48:31.240048 1490 tablet_service.cc:1505] Processing CreateTablet for tablet 97300d4eb8214d2bad6db432db47e129 (DEFAULT_TABLE table=default.Table_4102ac21c0a345da986f9f8f4d66a1b4 [id=c7994531a5904b1180d7ecabc1b39b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.243600 1489 tablet_service.cc:1505] Processing CreateTablet for tablet e4ae4b207f524de8a3629590bd9882db (DEFAULT_TABLE table=default.Table_f997c14171b3466f9c2c388c542735bd [id=dabcbcba9c0244ae86dc6fdae6325454]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.245724 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 7212e488655f4f5390eb683749f41994 (DEFAULT_TABLE table=default.Table_5bf3fd6ad2884dd8a4dcd772d793798d [id=e00cb3c037ed43d9afe9554146273f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.248111 1479 tablet_service.cc:1505] Processing CreateTablet for tablet c93eccd2a95b4a79a94d5e35f4414bb3 (DEFAULT_TABLE table=default.Table_5f7d0b8e722a457ab176bb08207e178c [id=2e682fd1a8d04539b3d22c8d7fde59d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.250566 1475 tablet_service.cc:1505] Processing CreateTablet for tablet 12bcf196c6b4429c9cade601782d27ef (DEFAULT_TABLE table=default.Table_c5daa131860c4f369dae0280b46a2165 [id=8492e7b523464e07abd051ee41cf63bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.252111 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ee9fb177b4a4dbc8fcd2c9f0b6ee699 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:31.252419 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet bc31e6c25ad14947af20abd766f12e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:02 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:31.254755 3860 tablet.cc:1722] T 1999b751e9344dcf81d26226c0589c8f P 43bec2f03fc849099e254f6335e9d532: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251024 19:48:31.257208 1330 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:54172 (request call id 187) took 1170 ms. Trace:
I20251024 19:48:31.257303 1330 rpcz_store.cc:276] 1024 19:48:30.086661 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:30.087116 (+ 455us) service_pool.cc:225] Handling call
1024 19:48:30.797724 (+710608us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x616000290854. stack: 00007f9883c039f6 00007f9883c037b0 00007f9882bdf3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f988fa98200 00007f989394466f 00007f988dd70c36 00007f988dd70965 00007f988ab9f27e 00007f988ab9e15e 00007f988aba4395 00007f988aba4cbf 00007f988aba4ae8 0000000000a8867c
1024 19:48:31.257181 (+459457us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":472061568}
I20251024 19:48:31.260653 1350 tablet_service.cc:1552] Processing DeleteTablet for tablet 799780bf376843499c659df4cd2b4aa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:31.260977 1334 tablet_service.cc:1552] Processing DeleteTablet for tablet 529580599bfa4800973b8540ab13d22a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:31.263623 1335 tablet_service.cc:1552] Processing DeleteTablet for tablet 786f0e890bb6410da9172eee32504be1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 19:48:30 UTC) from {username='slave'} at 127.0.0.1:54174
I20251024 19:48:31.264750 1350 tablet_service.cc:1505] Processing CreateTablet for tablet 97300d4eb8214d2bad6db432db47e129 (DEFAULT_TABLE table=default.Table_4102ac21c0a345da986f9f8f4d66a1b4 [id=c7994531a5904b1180d7ecabc1b39b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.265151 1350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97300d4eb8214d2bad6db432db47e129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.265596 1334 tablet_service.cc:1505] Processing CreateTablet for tablet 7212e488655f4f5390eb683749f41994 (DEFAULT_TABLE table=default.Table_5bf3fd6ad2884dd8a4dcd772d793798d [id=e00cb3c037ed43d9afe9554146273f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.265926 1334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7212e488655f4f5390eb683749f41994. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.266925 1346 tablet_service.cc:1505] Processing CreateTablet for tablet c93eccd2a95b4a79a94d5e35f4414bb3 (DEFAULT_TABLE table=default.Table_5f7d0b8e722a457ab176bb08207e178c [id=2e682fd1a8d04539b3d22c8d7fde59d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.267234 1346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c93eccd2a95b4a79a94d5e35f4414bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.268010 1480 tablet_service.cc:1505] Processing CreateTablet for tablet 0bd17c969314425799a12e2cb878d957 (DEFAULT_TABLE table=default.Table_df274196c7264f7aa9c5e59b7b50d657 [id=6a0cdea1153c4a9d957aa87f52d8c974]), partition=RANGE (key) PARTITION UNBOUNDED
W20251024 19:48:31.269632 1768 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.070s sys 0.133s
W20251024 19:48:31.269791 1768 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.612s user 0.070s sys 0.133s
I20251024 19:48:31.270099 1488 tablet_service.cc:1505] Processing CreateTablet for tablet 76a4c7ec8e514b26b6b270fdab8b9b5a (DEFAULT_TABLE table=default.Table_81498524d757401faaff766b9db57c27 [id=2613adda6a1c438bae1b3db87fb910d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.273165 1476 tablet_service.cc:1505] Processing CreateTablet for tablet 56e63d72f666493785eaa8ac46e71e91 (DEFAULT_TABLE table=default.Table_2fc89eee3d764c409f66027b6b878714 [id=4c8fa97560434affb57bc85e64dad190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.274446 1768 tablet_bootstrap.cc:492] T 8d1b6eaa353c4284a2b114b42437272c P 43bec2f03fc849099e254f6335e9d532: No bootstrap required, opened a new log
I20251024 19:48:31.274657 1768 ts_tablet_manager.cc:1403] T 8d1b6eaa353c4284a2b114b42437272c P 43bec2f03fc849099e254f6335e9d532: Time spent bootstrapping tablet: real 0.622s user 0.070s sys 0.139s
I20251024 19:48:31.275029 1478 tablet_service.cc:1505] Processing CreateTablet for tablet 8d00c2d71fe7416c80a690f7c48e03d1 (DEFAULT_TABLE table=default.Table_cc2d8128e49f486bbc9cf83bb4b112cc [id=ef8ef07b3ff047579e125b267b7de277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.276564 1335 tablet_service.cc:1505] Processing CreateTablet for tablet e4ae4b207f524de8a3629590bd9882db (DEFAULT_TABLE table=default.Table_f997c14171b3466f9c2c388c542735bd [id=dabcbcba9c0244ae86dc6fdae6325454]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.276950 1335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4ae4b207f524de8a3629590bd9882db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.278074 1484 tablet_service.cc:1505] Processing CreateTablet for tablet db4dd2ff754342359ffde52ec47c5e62 (DEFAULT_TABLE table=default.Table_2ca6f1a424034ecdb01cef12ae91756c [id=301745a621a043b883dab93342a76553]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.279085 1483 tablet_service.cc:1505] Processing CreateTablet for tablet 29f6d72182bb4f289e8f4b965da038b1 (DEFAULT_TABLE table=default.Table_0b75e1eb5e35490ab2947463e6788059 [id=4c5601b2d4c6479186929fd28ab240e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.279592 1481 tablet_service.cc:1505] Processing CreateTablet for tablet bd345b7c5f304004a908c99cb511e4d1 (DEFAULT_TABLE table=default.Table_353d0c63ddcc41d2aa242bb7e8f89f14 [id=a619e35185574efcb1f578d2e5610c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.280622 1338 tablet_service.cc:1505] Processing CreateTablet for tablet 12bcf196c6b4429c9cade601782d27ef (DEFAULT_TABLE table=default.Table_c5daa131860c4f369dae0280b46a2165 [id=8492e7b523464e07abd051ee41cf63bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.280946 1338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12bcf196c6b4429c9cade601782d27ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.281790 1339 tablet_service.cc:1505] Processing CreateTablet for tablet 76a4c7ec8e514b26b6b270fdab8b9b5a (DEFAULT_TABLE table=default.Table_81498524d757401faaff766b9db57c27 [id=2613adda6a1c438bae1b3db87fb910d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.282071 1339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a4c7ec8e514b26b6b270fdab8b9b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.280820 1342 tablet_service.cc:1505] Processing CreateTablet for tablet 0bd17c969314425799a12e2cb878d957 (DEFAULT_TABLE table=default.Table_df274196c7264f7aa9c5e59b7b50d657 [id=6a0cdea1153c4a9d957aa87f52d8c974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.282642 1342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd17c969314425799a12e2cb878d957. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.283354 1768 raft_consensus.cc:359] T 8d1b6eaa353c4284a2b114b42437272c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:31.283504 1768 raft_consensus.cc:385] T 8d1b6eaa353c4284a2b114b42437272c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 19:48:31.283547 1768 raft_consensus.cc:740] T 8d1b6eaa353c4284a2b114b42437272c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bec2f03fc849099e254f6335e9d532, State: Initialized, Role: FOLLOWER
I20251024 19:48:31.283704 1768 consensus_queue.cc:260] T 8d1b6eaa353c4284a2b114b42437272c P 43bec2f03fc849099e254f6335e9d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:31.284209 1768 ts_tablet_manager.cc:1434] T 8d1b6eaa353c4284a2b114b42437272c P 43bec2f03fc849099e254f6335e9d532: Time spent starting tablet: real 0.009s user 0.000s sys 0.000s
I20251024 19:48:31.284471 1768 tablet_bootstrap.cc:492] T aa08555dbe1e4ad8968c534510eca9ff P 43bec2f03fc849099e254f6335e9d532: Bootstrap starting.
I20251024 19:48:31.286052 1768 tablet_bootstrap.cc:654] T aa08555dbe1e4ad8968c534510eca9ff P 43bec2f03fc849099e254f6335e9d532: Neither blocks nor log segments found. Creating new log.
I20251024 19:48:31.286788 1347 tablet_service.cc:1505] Processing CreateTablet for tablet 29f6d72182bb4f289e8f4b965da038b1 (DEFAULT_TABLE table=default.Table_0b75e1eb5e35490ab2947463e6788059 [id=4c5601b2d4c6479186929fd28ab240e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.290988 1347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f6d72182bb4f289e8f4b965da038b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.287127 1340 tablet_service.cc:1505] Processing CreateTablet for tablet bd345b7c5f304004a908c99cb511e4d1 (DEFAULT_TABLE table=default.Table_353d0c63ddcc41d2aa242bb7e8f89f14 [id=a619e35185574efcb1f578d2e5610c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.288048 1349 tablet_service.cc:1505] Processing CreateTablet for tablet db4dd2ff754342359ffde52ec47c5e62 (DEFAULT_TABLE table=default.Table_2ca6f1a424034ecdb01cef12ae91756c [id=301745a621a043b883dab93342a76553]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.289809 1345 tablet_service.cc:1505] Processing CreateTablet for tablet 8d00c2d71fe7416c80a690f7c48e03d1 (DEFAULT_TABLE table=default.Table_cc2d8128e49f486bbc9cf83bb4b112cc [id=ef8ef07b3ff047579e125b267b7de277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.290077 1336 tablet_service.cc:1505] Processing CreateTablet for tablet 56e63d72f666493785eaa8ac46e71e91 (DEFAULT_TABLE table=default.Table_2fc89eee3d764c409f66027b6b878714 [id=4c8fa97560434affb57bc85e64dad190]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.294008 1340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd345b7c5f304004a908c99cb511e4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.294214 1349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db4dd2ff754342359ffde52ec47c5e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.297091 1486 tablet_service.cc:1505] Processing CreateTablet for tablet f7534929046a485bbf461ddd0b468ebb (DEFAULT_TABLE table=default.Table_862cb2dda48a4dbd954b5d852fb0e5fb [id=d5b82997e3bb400c8820bcc2b8d58f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.297767 1336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56e63d72f666493785eaa8ac46e71e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.298344 1345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d00c2d71fe7416c80a690f7c48e03d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.298977 1482 tablet_service.cc:1505] Processing CreateTablet for tablet cab80a02b1264bf1b4e40f459a467be3 (DEFAULT_TABLE table=default.Table_535b9a93bdda424a81bbdc6c08c3450f [id=9cb0b1b40eaf46f798e6f018727f8d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.301611 1491 tablet_service.cc:1505] Processing CreateTablet for tablet 79ad354b4f844055baaa997cee6a3d3f (DEFAULT_TABLE table=default.Table_7040850ac4da4483ad7cc6d41cc17eb4 [id=bd49e5c0642a4bd1b983f4c534887644]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.303748 1477 tablet_service.cc:1505] Processing CreateTablet for tablet 175a2aa798ec40e6a57a4b63be7e180b (DEFAULT_TABLE table=default.Table_b1945a1bef6a40fb8dd92f10bcb2ff10 [id=52bce2dfe0c543faa2fbc2b0592e4bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.202364 1776 ts_tablet_manager.cc:1950] T ab488e4971874a16b2e052c84c250777 P af3952de050140d383f6b0de7666fa97: Deleting consensus metadata
I20251024 19:48:31.206923 1618 tablet_service.cc:1505] Processing CreateTablet for tablet 97e529e139d2402fbabe1111bb95f0c4 (DEFAULT_TABLE table=default.Table_3bca07553d454224b50597235f74a30a [id=dbd60b012488487f8cffc49148bf9919]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.306103 1618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97e529e139d2402fbabe1111bb95f0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.306044 1474 tablet_service.cc:1505] Processing CreateTablet for tablet 80be570302354c2ba5be47f22ef5e083 (DEFAULT_TABLE table=default.Table_db05043847df4217a12a2524e5dc05ce [id=959edfd20d594810ba953638a3cdc7eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.307726 1472 tablet_service.cc:1505] Processing CreateTablet for tablet a8f7c5db8eb84671be28c5368c345f37 (DEFAULT_TABLE table=default.Table_d65da5a2f7214fdca166986046157c60 [id=7a8e293425f641daae6afc0de8c01caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.212195 1625 tablet_service.cc:1505] Processing CreateTablet for tablet 1955e14951df4cc295ea1041f8a1bce6 (DEFAULT_TABLE table=default.Table_2f47166e708545b68ae28fad4b98beb7 [id=4f7c1267bb5c4233a258baabc960b2c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.310019 1625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1955e14951df4cc295ea1041f8a1bce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.212195 1624 tablet_service.cc:1505] Processing CreateTablet for tablet 254df86cc6134c6680ff197fecc126de (DEFAULT_TABLE table=default.Table_0a49ed7967e3412d8bfd9c208fc8c67c [id=af99524888cf412292f801856c9a6358]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.313987 1624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 254df86cc6134c6680ff197fecc126de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.314028 1487 tablet_service.cc:1505] Processing CreateTablet for tablet 701c7094a6bc41bb87483d150e7d0727 (DEFAULT_TABLE table=default.Table_22c143aabac54b76afbf3cf41fac7d57 [id=cf1bf27deea242d392906a54987ef11b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.212905 1619 tablet_service.cc:1505] Processing CreateTablet for tablet c05da9c52bb44afc831d74b88a2a078b (DEFAULT_TABLE table=default.Table_153cb8026eb94425a5f0fc04046cd80a [id=32004498b1ef49ac975ee3fb5da334e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.314837 1619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c05da9c52bb44afc831d74b88a2a078b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.213238 1615 tablet_service.cc:1505] Processing CreateTablet for tablet 49e03fe24f5e43d38c4f0f45063554d4 (DEFAULT_TABLE table=default.Table_19a71f29de4849f5ad1d9e9741af2412 [id=70a75ac4105c4410ac64153f1b7b3a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.317899 1615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49e03fe24f5e43d38c4f0f45063554d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.318643 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1311) took 3172 ms. Trace:
I20251024 19:48:31.318727 1776 rpcz_store.cc:276] 1024 19:48:28.146305 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:28.146943 (+ 638us) service_pool.cc:225] Handling call
1024 19:48:31.034521 (+2887578us) ts_tablet_manager.cc:1151] Deleting tablet ab488e4971874a16b2e052c84c250777
1024 19:48:31.034530 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:31.166938 (+132408us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:31.180395 (+ 13457us) tablet_metadata.cc:641] Metadata flushed
1024 19:48:31.318614 (+138219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2887421}
I20251024 19:48:31.214207 1613 tablet_service.cc:1505] Processing CreateTablet for tablet d52923fb3ad141e99951c0d76c384c3d (DEFAULT_TABLE table=default.Table_6993ba61772248f4b2788ef4947af471 [id=562fdb5f53344a05bae7c056f6f11b08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.322042 1613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52923fb3ad141e99951c0d76c384c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.214419 1611 tablet_service.cc:1505] Processing CreateTablet for tablet 84a17140c301412abe0b2232287da909 (DEFAULT_TABLE table=default.Table_c929febd979f4b6f85071cd5852e90e6 [id=f7f131625dbb455ab0ad6f535f243062]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.325845 1611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84a17140c301412abe0b2232287da909. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.328567 1776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1313) took 3168 ms. Trace:
I20251024 19:48:31.328655 1776 rpcz_store.cc:276] 1024 19:48:28.160651 (+ 0us) service_pool.cc:168] Inserting onto call queue
1024 19:48:28.160771 (+ 120us) service_pool.cc:225] Handling call
1024 19:48:31.328416 (+3167645us) ts_tablet_manager.cc:1151] Deleting tablet f73069b9201449d08a4f7b246444e891
1024 19:48:31.328425 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1024 19:48:31.328548 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3167401}
I20251024 19:48:31.329190 1776 tablet_replica.cc:333] T 292c31e7d4ce411bb336a1405ca43ba5 P af3952de050140d383f6b0de7666fa97: stopping tablet replica
I20251024 19:48:31.215698 1622 tablet_service.cc:1505] Processing CreateTablet for tablet a856862373e4475ea81ec6c90deb3578 (DEFAULT_TABLE table=default.Table_f0f1e12cae86473ea751143cac835f76 [id=02be5e87772a45419b221edd4421a85f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.216342 1614 tablet_service.cc:1505] Processing CreateTablet for tablet e9384e55729c4d5eb073c754a172a52a (DEFAULT_TABLE table=default.Table_40837137d2a145828e6dcbd42e88e393 [id=a174648aa0f944d193fde7710520a5e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.217825 1623 tablet_service.cc:1505] Processing CreateTablet for tablet f57784ab866f4cd598c41e4fea6224a1 (DEFAULT_TABLE table=default.Table_ab5fc4c134d94ab489257b3a84b1b485 [id=88009a8264984d9196a91c5c1fef4d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.218135 1612 tablet_service.cc:1505] Processing CreateTablet for tablet be6a55e263ce4e9d85ddb2443e5a8426 (DEFAULT_TABLE table=default.Table_df4b29031df04d61ba08e9741f230fda [id=003f20c882314503a8334a05bc68b955]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.218398 1628 tablet_service.cc:1505] Processing CreateTablet for tablet 46311f0b92934ded971f5758a40fe43f (DEFAULT_TABLE table=default.Table_69274ddf02254478b87008336a851c38 [id=60402d110a0745528957bc79be951843]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.218691 1617 tablet_service.cc:1505] Processing CreateTablet for tablet 6e24fd1eb29f4560a7ccbc2f8a480b19 (DEFAULT_TABLE table=default.Table_37d5be312af34d6d920c4c7ad8acadc5 [id=ebd20c2af5954c398f2cf30a1b7dc037]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.329859 1622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a856862373e4475ea81ec6c90deb3578. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.330052 1623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f57784ab866f4cd598c41e4fea6224a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.219285 1616 tablet_service.cc:1505] Processing CreateTablet for tablet ee6ac4d03b1d48489ba07ed65eaf79ee (DEFAULT_TABLE table=default.Table_23ab85c60df54739b2f4245d7463f4b0 [id=43acfd2774ff4106a981b82f930a79b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.330395 1616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee6ac4d03b1d48489ba07ed65eaf79ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.219568 1621 tablet_service.cc:1505] Processing CreateTablet for tablet 5f89c5b39bf1429e8c5cfd9165a22b7e (DEFAULT_TABLE table=default.Table_92a5692084bf4d7c9caf7f90960e8f9b [id=c683f472ffba473c865e9a2f7838c63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 19:48:31.333894 1621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f89c5b39bf1429e8c5cfd9165a22b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.337489 1176 catalog_manager.cc:4985] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): tablet ab488e4971874a16b2e052c84c250777 (table default.Table_93cc576cc317428c88108b05bf8efc6d [id=c69469ba6a9745f6a66092a678a5861d]) successfully deleted
I20251024 19:48:31.337563 1612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be6a55e263ce4e9d85ddb2443e5a8426. 1 dirs total, 0 dirs full, 0 dirs failed
W20251024 19:48:31.338125 1176 catalog_manager.cc:4965] TS af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): delete failed for tablet f73069b9201449d08a4f7b246444e891 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f73069b9201449d08a4f7b246444e891 already in progress: creating tablet
I20251024 19:48:31.341050 1628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46311f0b92934ded971f5758a40fe43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.341838 1617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e24fd1eb29f4560a7ccbc2f8a480b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.345849 1614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9384e55729c4d5eb073c754a172a52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 19:48:31.347746 3845 raft_consensus.cc:493] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:31.347899 3845 raft_consensus.cc:515] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } }
I20251024 19:48:31.348537 3845 leader_election.cc:290] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 43bec2f03fc849099e254f6335e9d532 (127.0.252.193:36103)
I20251024 19:48:31.225947 3879 tablet.cc:1722] T 92d78642772d44179e67d75c51b9da4e P af3952de050140d383f6b0de7666fa97: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251024 19:48:31.349443 1645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3e9960b7821421aa23e7ef5389f894a" candidate_uuid: "10a9536207e343f188e0ef709081d04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3952de050140d383f6b0de7666fa97" is_pre_election: true
W20251024 19:48:31.350108 1444 leader_election.cc:343] T b3e9960b7821421aa23e7ef5389f894a P 10a9536207e343f188e0ef709081d04e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119): Illegal state: must be running to vote when last-logged opid is not known
I20251024 19:48:31.353689 1190 master_service.cc:954] ReplaceTablet: received request to replace tablet 0e3a8d735a1741cabcabd64964f172fd from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:31.355039 1184 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51416:
name: "default.Table_f3ff5150c2264528ab869400f8647a1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 19:48:31.355595 1184 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3ff5150c2264528ab869400f8647a1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 19:48:31.357785 3846 raft_consensus.cc:493] T 0797d14e9fd846788164747523df0e2c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 19:48:31.357955 3846 raft_consensus.cc:515] T 0797d14e9fd846788164747523df0e2c P 43bec2f03fc849099e254f6335e9d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af3952de050140d383f6b0de7666fa97" member_type: VOTER last_known_addr { host: "127.0.252.195" port: 34119 } } peers { permanent_uuid: "43bec2f03fc849099e254f6335e9d532" member_type: VOTER last_known_addr { host: "127.0.252.193" port: 36103 } } peers { permanent_uuid: "10a9536207e343f188e0ef709081d04e" member_type: VOTER last_known_addr { host: "127.0.252.194" port: 34045 } }
I20251024 19:48:31.358614 3846 leader_election.cc:290] T 0797d14e9fd846788164747523df0e2c P 43bec2f03fc849099e254f6335e9d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3952de050140d383f6b0de7666fa97 (127.0.252.195:34119), 10a9536207e343f188e0ef709081d04e (127.0.252.194:34045)
I20251024 19:48:31.360481 1183 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_097afacca75e46dfbfec9dd862b21a6c" } new_table_name: "default.Table_fe737a7e50574ff6a1555fe0a2b5ea5e" modify_external_catalogs: true
I20251024 19:48:31.362874 1189 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_692697818af34700ad79bbc08b8deb43" } new_table_name: "default.Table_4b7874b498224661a419170ede262025" modify_external_catalogs: true
I20251024 19:48:31.364707 1190 catalog_manager.cc:5936] T 00000000000000000000000000000000 P dc8f71a5b70749a4a3e3a599bcb3d22c: Sending DeleteTablet for 3 replicas of tablet 0e3a8d735a1741cabcabd64964f172fd
I20251024 19:48:31.365159 1190 catalog_manager.cc:6719] ReplaceTablet: tablet 0e3a8d735a1741cabcabd64964f172fd deleted and replaced by tablet 201c9dced06a44249d9a04a00b801c84
I20251024 19:48:31.365509 1191 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_7837f9d39d7e4658a8945b4ddd2cf236" } new_table_name: "default.Table_7decaaea5e0b4698af392f019d8d179d" modify_external_catalogs: true
I20251024 19:48:31.367825 1192 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_06fa0bf1c93d492d82dc0e412d9418b3" } modify_external_catalogs: true
I20251024 19:48:31.367975 1192 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51416:
table { table_name: "default.Table_06fa0bf1c93d492d82dc0e412d9418b3" } modify_external_catalogs: true
I20251024 19:48:31.370668 1186 master_service.cc:954] ReplaceTablet: received request to replace tablet 8af55c71122f4712ba1050e42f01ed83 from {username='slave'} at 127.0.0.1:51416
I20251024 19:48:31.375299 1614 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e3a8d735a1741cabcabd64964f172fd with delete_type TABLET_DATA_DELETED (replaced by tablet 201c9dced06a44249d9a04a00b801c84) from {username='slave'